Into the ORBIT for Time Series: Training Regimes for Foundation Models
Hongjie Xia, Yiding Liu, Yifan Hu, Peiyuan Liu, Zewei Dong
cs.LG, cs.AI
2026-08-13
Ant Group's Falcon-2.0 pairs a deliberately plain 585M encoder-only Transformer with ORBIT, a sampling paradigm that explicitly controls the effective pre-training distribution, and takes the best GIFT-Eval point-forecast MASE (0.6684) among 29 pretrained models.
Progress in time-series foundation models (TSFMs) has concentrated almost entirely on architecture: group attention, flow matching, serial-token prediction, MoE scaling. What actually determines what the model learns is how training data gets exposed: which datasets appear how often, with how much context, forecasting how far ahead, and how missing observations are handled at every stage. That layer has been barely studied.
Four coupled failure points: high-volume datasets dominate the optimization trajectory while under-represented domains starve; different sampling frequencies need different context ranges, so fixed contexts under-serve broad generalization; horizons vary across applications, and existing models often resort to horizon-specific schedules; missing values pervade real data and demand consistent treatment across normalization, tokenization, attention, and loss. MOMENT's sequential traversal plus sliding-window enumeration gives exposure proportional to window count; Moirai's length-aware weighting still favors datasets with greater aggregate length. The effective pre-training distribution is decided by corpus layout rather than by design.
Falcon-2.0 (Ant Group's international team, technical report, weights released) makes this layer the main battleground and keeps the architecture deliberately plain.
ORBIT (Omni-Range Bootstrap Incremental Training) has two halves.
Bootstrap Multi-Level Sampling constructs the distribution. At the corpus level, prescribed dataset weights are translated into a global training stream by a low-discrepancy greedy blending rule: each slot picks the dataset with the largest cumulative deficit against its target count. Independent categorical sampling matches proportions only in expectation; this rule keeps cumulative composition on target after every assignment. Within each dataset, a four-level stochastic procedure builds an offline sample index of five-tuples (record, target variable, context start, context end, horizon): records sampled uniformly, variables uniformly, the split point uniformly over feasible positions, then context start and horizon uniformly over their conditional ranges. The index is built once, cached, reproducible, and supports random access. Contrast with sliding-window enumeration, where adjacent windows overlap heavily and exposure follows window count; here every dimension is randomized and exposure follows design.
Omni-Range Incremental Training consumes it. Contexts are left-padded and targets right-padded to batch maxima, attention and loss masks exclude invalid positions, and short/long contexts coexist with short/long horizons in the same batch, in a single training stage. No Chronos-2-style capability refinement phase, no Timer-S1-style context-extension stage. Segments load on demand from memory-mapped storage.
The backbone is a simplified Chronos-2 univariate encoder: missingness-aware triple-channel patch tokens (temporal, value, indicator), reversible arcsinh instance normalization computed over observed values only, a bidirectional encoder with a REG token, future query patches in the same sequence, and a quantile head emitting 21 quantiles for all future patches at once. No group attention; variables are independent. 32 layers, d=1024, 585M parameters, trained for 1M steps with Megatron-LM on B200 clusters.
The second contribution, Rank-Guided Cross-Depth Alignment, pulls the shallow layer (block 1) toward the deep layer (block 31) with a stop-gradient token-wise cosine objective. A perturbation argument shows that sufficiently small alignment error prevents the shallow representation from dropping non-negligible spectral modes present in the deep one. Training-only, zero inference cost.
On GIFT-Eval (97 dataset-frequency-horizon configurations), Falcon-2.0 takes the lowest normalized MASE, 0.6684, and the best mean rank, 7.81, among 29 pretrained models, ahead of STRIDE+Timer-S1 at 0.6744. Probabilistically its CRPS of 0.4843 ranks seventh, behind STRIDE+Chronos-2 at 0.4544.
On fev-bench (100 tasks), its MASE of 0.6459 sits 0.3% behind TimesFM-2.5 (0.6438), tied with Chronos-2, but its WQL of 0.4842 is the best overall, and no baseline beats it on both MASE and WQL simultaneously.
The ablations carry the paper:
| Ablation | GIFT-Eval MASE change | Takeaway |
| Sliding-window to Bootstrap sampling | down 11.7% | The sampling rule is the largest single factor |
| Fixed horizon to joint sampling | down 6.1% | Horizon sampling matters more than context sampling |
| Fixed context to joint sampling | down 2.9% | Useful but secondary |
| Remove parallel patch prediction | up 8.0% | Largest architectural contributor |
Notably, among the four sliding-window variants, fixing both lengths beats sampling either one, yet all four lose to Bootstrap stochastic sampling across the board. Varying lengths alone is not enough; removing adjacent-window redundancy and randomizing record and variable selection is where the gain lives.
Scaling is mild: from 75M to 585M, GIFT-Eval MASE falls only 2.3%, fev-bench 4.2%. Metrics improve through the full 1M steps without reversal.
The message to the TSFM field is blunt: a carefully tuned architecture can lose to a carefully designed sampler. Falcon-2.0's backbone is essentially a reduced Chronos-2 recipe; the leaderboard position comes from data-exposure control. For teams building time-series pretraining, ORBIT is directly copyable engineering: offline indices, greedy blending, single-stage mixed-length training without multi-stage scheduling complexity.
For users, an open 585M model leading point forecasting on both leaderboards raises the price-performance bar for zero-shot forecasting again. With no model dominating both point and probabilistic metrics, Falcon-2.0 occupies the Pareto position of being near the front on both.
Probabilistic performance is not won: CRPS ranks seventh on GIFT-Eval, it clearly trails Toto-2.0-2.5B on multivariate tasks (0.476 vs 0.446) and long horizons. The 46 fev-bench tasks with known-future covariates are a structural weakness, MASE 0.652 versus Chronos-2's 0.621, because the autoregressive interface does not ingest future features. That is an architectural gap, not a tuning issue.
Read the technical report with discounts: sampling ablations run at the single 585M size; the Rank-Guided alignment objective gets no independent ablation number in the main text, so its contribution is unclear; the sliding-window baselines are the authors' reimplementations, differing in pipeline details from original papers; the pre-training corpus spans seven domains with statistics only in an appendix, and how the domain-aware weights were set is not stated; the B200-cluster training cost is undisclosed.
Also note the ECON/Fin weakness on point forecasts (Toto-2.0 at 0.739 versus Falcon-2.0 at 0.785); finance users should be cautious.