See requirements.txt
- test_data_parallelism.py
- Nearly identical to
Accelerate
's example but using a larger model and changing the default batch_size settings. - Launch dual GPU training on single node with mixed precision:
python -m torch.distributed.run --nproc_per_node 2 --use_env test_data_parallelism.py --fp16=True
- Nearly identical to
- test_model_parallelism.py
- Data parallelism & model parallelism without relying on
Accelerate
library. - Only supports single node dual GPU training without mixed precision.
- Launch dual GPU training on single node without mixed precision:
python test_model_parallelism.py
- Data parallelism & model parallelism without relying on
Apache License 2.0