AWS Reveals Advanced SFT Strategies: More Data Is Not Always Better
AWS ML Blog · rss · 2026-08-27
The AWS ML Blog published Part 2 of their SFT data preparation guide, focusing on advanced strategies. The post highlights that SFT does not follow pre-training power-law scaling; more data does not guarantee better performance. Key points include:
- Data Volume: A typical SFT task requires roughly 2,000 high-quality samples. Use learning curve analysis to find the performance saturation point.
- Subset Selection: Intelligent data selection (e.g., DEITA, AlpaGasus) often outperforms full-dataset training under fixed compute and reduces catastrophic forgetting.
- Data Augmentation: When data is scarce, expand it via distillation from stronger models, self-generation and filtering, or amplifying expert annotations.
- Workflow: Evaluate data gains by saving checkpoints during a single training run. If gains diminish, stop training or collect targeted data.
More from coding & agent
- Developer open-sources Gemini 3.5 powered dictation app — ammaar · 2026-08-27
- Developer open-sources Wispr Flow-like app powered by Gemini 3.5 Transcribe — ammaar · 2026-08-27
- Offline Wikipedia RAG via 100-line Bash script works on local LLMs — mantisalt · 2026-08-27
- Hermes LCM: Compresses 173k Tokens to 2.8k with Full Retrievability — iamrobotbear · 2026-08-27
- Google releases Gemini 3.5 Transcribe: filters filler words and understands codebase context — AI_Andrew · 2026-08-27
- Why AI agents need stablecoins: traditional settlement takes T+2 to T+60 days — kleffew94 · 2026-08-27