LLM stock-news scores were a coin flip, so this dev rebuilt labels from market reactions
Fun_Water2230 · reddit · 2026-09-24
The dev behind deepmoney audited his LLM news screener for Chinese A-share headlines and found great prose with zero signal: AUC 0.59/0.47 across two months, and a self-rated "stock relevance" score with rank IC of −0.094 against 20-day excess returns (sign flipped).
He rebuilt everything around one rule: the market is the only judge.
- Label factory: measures minute-bar reactions of stock baskets at publish time vs 500 random same-minute baskets, strictly as-of with no look-ahead
- Placebo tests: "major reaction" labels hit 3.2% vs 0.3–0.9% placebo; failing label ideas (e.g. market-wide layer) were dropped
- Counterintuitive finding: news the old pipeline never extracted reacted as strongly as extracted news — treat as missing labels, not negatives
- Jev-inspired model: Qwen3.8-27B + LoRA, lmhead replaced with ordinal cumulative-link heads for free early exit; one prefill yields 5 answers (30-min reaction, T+1, volume burst, reprints, follow-up coverage); market state as soft tokens; ordinal NLL instead of RL for calibration
Honest caveats: 27B not fully trained yet (smoke test on Qwen3-0.6B only); baseline to beat is a text-blind GBM at 0.55–0.57 AUC on price and 0.68–0.70 on volume. MIT-licensed code plus a 283k-item labeled Chinese news dataset (non-commercial research).
More from coding & agent
- Claude Opus 5.5 draws a 40-second 'how browsers work' animation frame by frame in JS — addyosmani · 2026-09-24
- One prompt unlocks more creativity from Claude Opus 5.5: "take all the time you need" — RileyRalmuto · 2026-09-24
- Guillaume Verdon calls for standard flight-booking and food-ordering benchmarks for agents — beffjezos · 2026-09-24
- Open-source video cloning skill turns reference videos into editable, locally rendered projects — oran_ge · 2026-09-24
- Magnitude: open-source tool profiles your PC and auto-tunes the best local model for agents — anselm · 2026-09-24
- Putting a Decision Model Like JEV in Production: Thresholds Are Margins, Not Accuracy — sven_ai · 2026-09-24