- Omar Zoloev
- Konstantin Zorin
- Alex Boriskin
- Ivan Lisitsyn
- Nikita Vakhrameev
This repository contains the implementation and experiments for the ReLoRA paper.
paper: ReLoRA: High-Rank Training Through Low-Rank Updates
-
Data:
.../src/data- Contains all the datasets used for the experiments.
-
Modules:
.../src/modules- Contains all the modules and scripts for the implementation of ReLoRA.
The dataset and metric used for this project were taken from a competition on Kaggle Automated Essay Scoring 2.0
import sklearn
from sklearn.metrics import cohen_kappa_score
def quadratic_weighted_kappa(y_pred, y_true):
return cohen_kappa_score(
y_true.astype(int),
y_pred.clip(0, 5).round(0),
weights='quadratic',
)| Metric / Optimizer | AdamW 3 epoch | AdamW Without reset optimizer | Adagrad 7 epoch | AdamW reset optimizer ½ weights | AdamW reset optimizer ¼ weights |
|---|---|---|---|---|---|
| QWK | 0.741 | 0.690 | 0.721 | 0.716 | 0.730 |
| MSE Loss per epoch | 0.500 | 0.601 | 0.550 | 0.540 | 0.531 |