accelerate

Hugging Face Accelerate is a library that abstracts away the complexity of running PyTorch training across different hardware configurations — enabling the same training script to run on a single CPU, single GPU, multi-GPU, multi-node cluster, or TPU without rewriting distributed training boilerplate, by wrapping model, optimizer, and dataloader with a single `accelerator.prepare()` call that handles device placement, gradient synchronization, and mixed precision automatically.

What Is Accelerate?

• Definition: A Python library by Hugging Face that provides a minimal abstraction layer over PyTorch's distributed training capabilities — handling `torch.distributed`, `DataParallel`, `FullyShardedDataParallel` (FSDP), DeepSpeed, and TPU XLA behind a unified interface.
• The Problem: Writing PyTorch code that works on both a laptop and a multi-GPU cluster is hard — you need `torch.distributed.launch`, `local_rank` management, gradient accumulation, mixed precision scaling, and device-specific code paths. Accelerate handles all of this.
• Minimal Code Changes: Add 4 lines to any PyTorch training script — `accelerator = Accelerator()`, `model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)`, replace `loss.backward()` with `accelerator.backward(loss)`. Done.
• Configuration-Driven: Run `accelerate config` once to set up your environment (number of GPUs, mixed precision, DeepSpeed stage) — then `accelerate launch train.py` runs your script with the configured distributed strategy.

Key Features

• Hardware Agnostic: The same script runs on CPU, single GPU, multi-GPU (DDP), multi-node, TPU, and Apple Silicon — Accelerate detects the hardware and applies the correct distributed strategy.
• Mixed Precision: Automatic FP16/BF16 mixed precision training — `Accelerator(mixed_precision="bf16")` enables mixed precision with no other code changes.
• DeepSpeed Integration: Full DeepSpeed ZeRO Stage 1/2/3 support — configure via `accelerate config` or a DeepSpeed config JSON, no DeepSpeed-specific code in your training script.
• FSDP Support: PyTorch FullyShardedDataParallel for training models that don't fit on a single GPU — shard model parameters, gradients, and optimizer states across GPUs.
• Gradient Accumulation: `accelerator.accumulate(model)` handles gradient accumulation across steps — essential for simulating large batch sizes on limited GPU memory.
• Big Model Inference: `accelerate` can load models larger than GPU memory using device_map="auto" — automatically splitting model layers across multiple GPUs or offloading to CPU/disk.

Accelerate vs Alternatives

Hugging Face Accelerate is the "write once, run anywhere" solution for PyTorch distributed training — adding just 4 lines of code to make any training script hardware-agnostic, with seamless DeepSpeed, FSDP, and mixed precision support that eliminates the distributed training boilerplate that traditionally consumes days of engineering effort.