GIFT-Eval-CTX: A Benchmark For Contextual Forecasting Model Evaluation
Xu Liu, Hanshu Yan, Taha Aksu, Quang Pham, Zhongzheng Qiao, Jun Hao Liew, Silvio Savarese, Doyen Sahoo, Junnan Li
cs.LG, cs.AI
2026-10-13
Salesforce builds 10,000 hard GIFT-Eval windows with effects from the realized future. GPT-5.6 Sol relative MASE falls from 0.953 to 0.565 (40.7%); MoiraiAgent hits 0.497.
Time-series foundation models take numeric history. A heatwave, a planned road closure, or an expert demand scenario is known before it appears in the series, and the usual interface cannot accept it.
Text-paired benchmarks such as Context is Key, Time-MMD, and FinTexTS often share three gaps: narrow domain coverage, no shared history-future split, and text that may say nothing about the future. A flat score then cannot separate unused context from empty context.
GIFT-Eval-CTX tests the second half only. Reading raw sources is left to experts and frontier LLMs. Context is taken from the future that already occurred and stored as structured effects, a checked scenario sheet. The measured skill is drawing an accurate trajectory once that sheet is in hand.
Windows come from GIFT-Eval held-out test regions. Multivariate series are split into univariate targets, and nothing is cut from train or validation. Futures shorter than 24 steps are dropped. Released horizons run from 30 to 900 steps. History is clipped to 10, 5, or 3 times the horizon for short, medium, and long settings.
A window must also be hard. Leading TSFMs forecast it, and rMAE is mean absolute error on the future divided by that future's range. Every model must score at least 0.10, so the error is at least a tenth of the future amplitude. Retained windows on one series do not overlap. The set has 10,000 instances, 20 datasets, 6 frequencies, and 78 dataset-frequency-term groups: 6,559 short, 2,135 medium, 1,306 long.
Effects are key-value fields, so wording stays out of the comparison. Macro covers the full horizon: one or two trend segments with direction, shape, and durations summing to 1; seasonality as a period and waveform, which may be empty; and a min and max normalized by the history range, allowed to fall outside 0 to 100. Micro is a closed interval the reference forecast missed: a deviation that recovers, a level jump that sticks, or a change in the periodic pattern. Code computes intensity from the raw series. There are 23,476 Micro effects, 2.35 per instance: 19,134 excursions, 3,117 seasonal changes, and 1,225 mean shifts.
Macro and Micro agents both see the true future. Validators recompute the value range and return the numbers when a proposal fails. Micro search uses the pointwise average of strong TSFM forecasts and proposes effects only where that residual is large. A critic reads the raw curve before seeing any proposal, then keeps, drops, or edits each interval. People checked a random subset. No review rate or error rate is reported. The paper states that generative models produced the annotations and that the authors inspected them.
The metric is MASE: absolute error divided by the in-sample error of a seasonal naive forecast. Within each of 78 groups the sample mean is divided by that group's seasonal naive, and the ratios are combined by a geometric mean. Seasonal naive scores 1.0. Lower is better. History-only baselines are Chronos-2, Toto-2, Moirai-2, and TimesFM-3, each returning a univariate median. GPT-5.6 Sol and Claude Opus 4.8 run with and without effects, with a 16,000-token output cap and medium reasoning effort. MoiraiAgent is GPT-5.6 Sol with a Moirai-2 forecast tool and a code tool, at most 10 steps. It edits an exact-length trajectory. The direct prompt also states the true min and max in raw units and tells the model to stay inside that interval.
On history alone both frontier LLMs lose to every TSFM. Aggregate relative MASE is 0.780 for TimesFM-3, 0.782 for Moirai-2, 0.795 for Chronos-2, 0.798 for Toto-2, 0.953 for GPT-5.6 Sol, and 1.109 for Claude Opus 4.8. Short horizons still beat seasonal naive (0.851 and 0.967 versus 1.0). At medium and long horizons GPT-5.6 Sol is 7.1% and 4.9% worse than seasonal naive; Claude is 32.5% and 20.6% worse.
| Method | Input | Short | Medium | Long | Overall |
| Seasonal naive | History | 1.000 | 1.000 | 1.000 | 1.000 |
| TimesFM-3 | History | 0.750 | 0.795 | 0.823 | 0.780 |
| Moirai-2 | History | 0.740 | 0.798 | 0.852 | 0.782 |
| Chronos-2 | History | 0.764 | 0.803 | 0.848 | 0.795 |
| Toto-2 | History | 0.765 | 0.811 | 0.851 | 0.798 |
| GPT-5.6 Sol | History | 0.851 | 1.071 | 1.049 | 0.953 |
| GPT-5.6 Sol | History + effects | 0.417 | 0.727 | 0.782 | 0.565 |
| Claude Opus 4.8 | History | 0.967 | 1.325 | 1.206 | 1.109 |
| Claude Opus 4.8 | History + effects | 0.538 | 1.077 | 1.147 | 0.781 |
| MoiraiAgent | History + effects | 0.398 | 0.623 | 0.603 | 0.497 |
Effects move GPT-5.6 Sol from 0.953 to 0.565, a 40.7% drop, and Claude from 1.109 to 0.781, a 29.6% drop. Gains shrink as the horizon grows. With effects, GPT-5.6 Sol beats history-only TSFMs at every horizon. Claude wins only on short horizons, 0.538 versus Moirai-2 at 0.740; medium 1.077 and long 1.147 stay worse than the TSFMs. MoiraiAgent leads overall at 0.497 and leads every horizon, 36.3% under TimesFM-3. Against GPT-5.6 Sol given the same effects, its margin is 4.6% short, 14.3% medium, and 22.9% long. Tools matter more as the horizon gets longer.
Macro alone, Micro alone, and both score 0.780, 0.650, and 0.565 for GPT-5.6 Sol, and 0.711, 0.549, and 0.497 for MoiraiAgent. The agent's history-only line is Moirai-2 at 0.782. Micro carries most of the gain. Macro alone fails at medium and long range: MoiraiAgent scores 0.802 and 0.858, slightly worse than history. Combined with Micro it is better at every horizon. One global description still matches many trajectories, and that slack grows with the horizon.
Mean shifts are the effects models actually execute. MASE falls to about one third, 0.890 to 0.294 and 0.780 to 0.283. Excursions drop about 29% to 31%. Seasonal changes drop 6% to 9%. On long horizons a seasonal-change note makes GPT-5.6 Sol worse, 1.070 to 1.119, while MoiraiAgent improves, 0.826 to 0.755. A level jump is easy to draw. A period change has to be applied across many cycles at once.
Inside labeled intervals, GPT-5.6 Sol goes from 1.011 to 0.496 (a 50.9% cut) and MoiraiAgent from 0.876 to 0.444 (49.3%). Outside those intervals the scores still fall, 0.890 to 0.802 and 0.687 to 0.652. One unlabeled gap between events falls from 0.261 to 0.124. On another series the cycle is preserved and MASE rises from 0.933 to 1.052. Keeping the shape and improving the score can come apart.
Exact output length on medium and long horizons is 38.9% and 40.5% for GPT-5.6 Sol, and 2.3% and 1.2% for Claude. The rest is truncated or padded. MoiraiAgent is exact on every sample because Moirai-2 already returned the right length. Within five history-normalized points, level-range and full-intensity adherence are 97.41% and 81.57% for GPT-5.6 Sol, 59.83% and 40.79% for Claude, and 99.98% and 92.73% for MoiraiAgent.
A 40.7% drop means a frontier LLM can execute a structured scenario. At short range, 0.417 already beats history-only Moirai-2 at 0.740. At long range, direct generation loses length and seasonality together, and editing a TSFM draft is the setup that holds: the agent margin grows to 22.9%. The forecasting backbone is unchanged.
An interface can stop at Macro and Micro keys. Mean shifts and excursions are the families to wire up first. Seasonal changes still return little.
Effects are mined from the future being scored. The test asks how much of a correct sheet can be reconstructed. Extraction from news is out of scope, and so is a wrong sheet. The prompt also supplies the true min and max in raw units and requires every point to land inside, while intensities come from the ground truth. A large share of the 40.7% is execution of those numbers. An open forecast does not arrive with this much of the answer filled in.
Windows with rMAE under 0.10 were removed, so a TSFM near 0.78 is not a full GIFT-Eval leaderboard result. Long horizons keep only three times the horizon of history, and multivariate series are split apart. The manual check has no published error rate. Micro intervals sit where this generation of TSFMs is wrong, so a new reference model moves both the hard set and the labels.
Code edits place the requested intensity and then flatten local motion into plateaus or straight lines. The two illustrated samples improve from 0.311 to 0.114 and from 0.389 to 0.224, and the prose and the figure disagree on which dataset receives which pair. The score can rise while the path looks less like the series. Claude matches the requested length on only about 2.3% and 1.2% of medium and long outputs, and the paper does not separate truncation from padding.