LoRA Training Test: Gradient Accumulation Is Not a Time-Zero Game
traceml-ai · reddit · 2026-08-19
Experiments with Qwen3-1.7B using TRL and LoRA reveal significant time differences for gradient accumulation strategies (1×4, 2×2, 4×1) despite identical effective batch sizes.
Results:
- T4: 4×1 is 17% faster than 1×4.
- L4: Difference is 41%, with 2×2 slightly faster than 4×1.
Analysis:
- Effective batch is an optimization knob, but physical batch decides the execution shape the GPU receives.
- 1×4 means four small forward/backward passes; 4×1 means one large pass. The samples reaching the optimizer are the same, but the GPU work is not.
- Performance may not be linear as physical batch increases.
Conclusion: Treat effective batch (optimization behavior) and physical batch (memory/speed) as separate choices. Start with the largest physical batch that fits.
Related event: LoRA Benchmarks: Larger Batch Beats Gradient Accumulation by 17%(2 posts)→
More from Infra
- Distributed Locking & TLA+ Verification at Modal — tokenbender · 2026-08-19
- Alchemy Adds Fly.io Machines API Provider with Resource Bindings — samgoodwin89 · 2026-08-19
- New tool cargo-bsize helps analyze and reduce Rust binary size — charliermarsh · 2026-08-19
- Open MAX and Form Open Alliance to Unify NVIDIA, AMD, Trainium, Google TPU, and More — clattner_llvm · 2026-08-19
- Higgsfield Chooses Together AI for Inference on Dedicated Containers — togethercompute · 2026-08-19
- ModCon '26 wraps: Mojo is now open source and heterogeneous compute has a real software platform — clattner_llvm · 2026-08-19