E2A-Bench: Benchmarking Evidence-to-Action Reliability in Financial Chart Reasoning
Xiaoya Wang, Yutong Xu, Junjie Wang
EMNLP Findings
cs.CL
2026-09-13
Tsinghua's 969-query HS300 chart bench finds Gemma-4-E2B at 6.4% coverage and 5.1% NDR despite the lowest hallucination, while Qwen2.5-VL-3B leads at 32.1% NDR.
In financial chart reasoning, a VLM is asked to read candlesticks, moving averages, and volume, then emit BUY, SELL, HOLD, or UNCERTAIN plus a confidence score. Most existing tests stop at "is this sentence visually supported" or "is the final answer right." Both views have blind spots. Coverage blindness: hallucination scores reward models that simply refuse to take a direction. Chain blindness: the test never checks whether evidence survives from chart facts through rationale and confidence into the action.
A model can look safe by flooding HOLD and suppressing unsupported claims, while producing almost nothing actionable. E2A-Bench, from Tsinghua and Jinan University, changes the evaluation unit to coverage-aware evidence-to-action reliability. It does not forecast returns. It audits whether the chain holds.
The benchmark covers 323 HS300 constituents. Each query uses a 30-trading-day OHLCV window under three inputs (text-only, chart-only, multimodal), for 969 queries. Ten deterministic evidence anchors per window yield 3,230 facts for claim checks and evidence-strength estimates. An audit of 200 random chart-label pairs found no mismatch between rendered charts and OHLCV labels.
The model writes a technical analysis, a rationale, an action, and a 1-5 confidence. Scoring splits into four links:
The headline metric is NDR = coverage × (1−UCRdir) × (1−RCI) × (1−ECI). The product is a necessary-condition score: absence, unsupported claims, self-contradiction, or overconfidence under weak evidence each zero out reliability. NDR is not PnL, and it does not assume statistical independence. The main setting is θ=0, so every directional action counts toward coverage; θ>0 is a stricter robustness check.
Twenty VLMs: Gemini 3.1 Pro/Flash, GPT-5.5/5.4/5.4-mini, Claude Sonnet 4.6; Qwen2.5-VL, Qwen3-VL, Qwen3.5, Gemma-4; and finance-tuned PyFi, Amsi-fin-o1, FinLLaVA.
| Model | Coverage | UCRall | NDR |
| Qwen2.5-VL-3B-Instruct | 82.5% | 53.1% | 32.1% |
| Qwen2.5-VL-7B-Instruct | 53.9% | 47.9% | 25.0% |
| PyFi-QwenVL-3B | 62.4% | 40.5% | 25.3% |
| Gemini-3.1-Pro | 59.0% | 70.0% | 16.8% |
| Gemma-4-E2B-It | 6.4% | 39.9% | 5.1% |
Gemma-4-E2B-It posts the lowest UCRall at 39.9%, yet only 6.4% directional coverage and 5.1% NDR. The small open model Qwen2.5-VL-3B leads at 32.1% NDR with 82.5% coverage. Proprietary systems are more conservative; the best of them, Gemini-3.1-Pro, reaches only 16.8% NDR.
Modality effects are model-dependent. Averaged over all 20 models, chart-only NDR is 19.2%, above text-only 14.5% and multimodal 15.6%. Qwen3.5-4B peaks at 37.9% on text; Gemma-4-31B peaks at 29.4% on charts. Multimodal input is not a stable win.
Financial fine-tuning does not uniformly reduce risk. Amsi-fin-o1 gains 1.5 NDR points; the two PyFi models drop 6.8 and 7.2 points. Across three strict base-to-tuned pairs, the BUY:SELL ratio grows 4.21-4.68× (1.41→6.10, 0.85→4.00, 1.84→7.73). Of 2,907 paired responses, 455 shift from HOLD/UNCERTAIN/unparseable to BUY, while matched evidence claims fall about 34-40%. Oracle-aided verification on charts can cut unsupported claims and still collapse actionable coverage, with a negative mean ΔNDR.
Qwen3.5 is not monotonic on chart-only input: NDR peaks at 2B (24.1), falls at 4B (12.6), recovers at 9B (18.1), and drops again at 27B (13.1). UCR rises from 57.0 to 71.0. HOLD and UNCERTAIN make up 63.6% of parsed actions, which is why coverage-aware scoring exists. In a blinded pilot, NDR had the strongest correlation with holistic human reliability judgments (ρ=0.466, p=0.009).
If you score financial VLMs on hallucination alone, refusal looks like reliability. E2A-Bench splits coverage from chain quality, and it is a practical audit for whether fine-tuning shoved the model onto one-sided BUY. It is not investment advice. A high NDR does not mean the system makes money.
For benchmark design this is a modest, pointed shift: move the unit of evaluation from claims or answers to the full evidence-to-action chain. Anchors and action schemas are swappable; the protocol is not locked to A-share technical analysis.
The snapshot is HS300 names on a fixed date. It does not span bull, bear, or sideways regimes, or other asset classes. Deterministic anchors and pattern detectors keep scoring auditable, and they will miss rare paraphrases and niche jargon. The 0.466 correlation with human judgments is a signal, not a gold standard. The paper reports oracle-aided verification as coverage-collapsing with negative mean ΔNDR, without a full per-model table in the main text. Fine-tuning corpora are not fully public, so the BUY tilt is descriptive, not causal.