Synthetic Data Tuning for a Small Time-Series Forecaster Stalls at ~0.2% Gains
tatjr13 · reddit · 2026-10-15
A Reddit practitioner asks for help: with a fixed pretrained small transformer forecaster and a short fixed training budget, only the synthetic data generator is tunable, yet the best lever (an extra stacked channel on 6% of records) yields just +0.2% across 6 seeds vs. a 1% target on a rolling real-world sensor eval.
Key findings: more such records hurt — structure matters, not channel content; gains don't transfer to a different starting checkpoint; the reference generator's edge concentrates at the longest horizon and fades as eval data rolls forward.
Open questions include: which data-side levers (curriculum, mixture weights, horizon-targeted data, augmentations) have moved error >1% under warm-start and tiny budgets; synthetic priors for TS foundation models beyond KernelSynth/ForecastPFN that survive distribution shift; how to quickly find weak slices of a rolling eval set; and whether a data-only lever is fundamentally capped here.
More from Research
- NCM: new text generation paradigm up to 31x faster than diffusion LMs with better quality — simonguozirui · 2026-10-15
- AlphaNIST: one RL policy solves all 20 NIST assembly tasks with 98.2% success in simulation — hbouammar · 2026-10-15
- Neural field planning for hybrid manufacturing cuts estimated build time 68.3% — jiqizhixin · 2026-10-15
- Tev-Gym: 1,170 dense-reward RL environments push long-horizon agentic training — RulinShao · 2026-10-15
- Claude systematically debunks the speculative EDAR-selenium hypothesis about East Asians — teortaxesTex · 2026-10-15
- Microsoft's AutoEnvScaling lets agents build their own RL environments, lifting Qwen3 Terminal-Bench from 41.1 to 53.3 — omarsar0 · 2026-10-15