@muellerzr the usage of the .data will greatly slow down pytorch/xla on nightly, we were hoping we can fix this issue before the next release.

The documentation is not available anymore as the PR was closed or merged.

This is a very old and deprecated implementation since it doesn't even follow the AdamW algorithm exactly. One should use torch.optim.AdamW instead, which also has a fused version since pt-2.0.0 which is almost as fast as apex's fused AdamW. So really you shouldn't be using this version anyway.

The only reason it was kept is for BC for those who rely on exact results remaining exact after new transformers versions are released, otherwise we would have just replaced it with torch.optim.AdamW in the first place.

p.s. no objections though to making it better...

@stas00 Thanks for the reply. How about the adafactor then?

oh, sorry, I didn't see it was Adafactor too. It's hard to see from the diff as it doesn't show the class names.

This Adafactor is being used for sure, but its implementation is super old as well. So certainly it'd be a blessing to bring it up to more modern code standard.

@stas00 Do you mind give this pr a review? Thanks.

Thanks for your PR. Just to be sure though, is this all going to work with PyTorch 1.8+? 1.8 is the minimum version we offically support at the moment (for a couple more weeks at least, then 1.9 starting mid-June).

I'm almost 100% sure it is the case. the whole direct .data usage deprecation is a few years old at least.

Let me quickly test it with pt-1.8

$ pytest tests/optimization/test_optimization.py -k test_adafactor
========================================================== test session starts ===========================================================
platform linux -- Python 3.8.8, pytest-7.3.1, pluggy-0.13.1
rootdir: /mnt/nvme0/code/huggingface/transformers-master
configfile: setup.cfg
plugins: timeout-1.4.2, typeguard-2.12.1, flakefinder-1.0.0, forked-1.3.0, monitor-1.6.0, hypothesis-6.47.0, instafail-0.4.2, xdist-2.2.1
collected 3 items / 2 deselected / 1 selected

tests/optimization/test_optimization.py .                                                                                          [100%]

================================================================= PASSES =================================================================
======================================================== short test summary info =========================================================
PASSED tests/optimization/test_optimization.py::OptimizationTest::test_adafactor
============================================== 1 passed, 2 deselected, 4 warnings in 0.16s ===============================================

$ pt-ver
pt=1.8.2, cuda=11.1, nccl=2708

At least the Adafactor test that we have is passing.

Thanks @sgugger and @stas00 for reviewing the changes.