Alnie Seed 9B model's benchmarks questioned: mismatched numbers, FP4 speed vs FP16 scores
cneuralnetwork · x · 2026-11-08
169Pi launched Alpie Seed, a 9B multimodal reasoning model, claiming 82.54% MMLU-Pro, 84.32% GPQA Diamond, 72.99% LiveCodeBench v6, and 61.20% SWE-Bench Verified. @cneuralnetwork read the technical report and found discrepancies:
1) Benchmark numbers in the launch tweet don't match the technical report (e.g., Qwen3.5-9B MMLU-Pro 80.0 on website vs 82.5 in report; Ministral 3 8B GPQA 38.6 vs 66.8).
2) The report admits comparison models weren't re-evaluated under identical conditions — with such small deltas, statistical significance is questionable.
3) A 5GB model allegedly needs only 1GB GPU VRAM, with no mention of quantization.
4) Speed tests ran at FP4 while accuracy results are FP16 — if speed/memory is the pitch, report FP4 results.
5) Small positive gains can't confirm SFT/GRPO helped without more ablations.
The critic still praises the model's engineering and post-training choices.
More from Models
- OpenAI Daybreak Blue lockout: hardware key swap triggers 403 verification failure — Formal_Kick2995 · 2026-11-08
- Google reportedly blocking Gemini subscription use in third-party coding harnesses, flagging accounts for ToS violations — PixelCactus1 · 2026-11-08
- Experiment: Claude commits to answers more firmly than ChatGPT on 29 of 34 science questions — AymericRoucher · 2026-11-08
- Claude Artifacts suddenly blocks writes, users asked to grant full network egress — KeepStandardVoice · 2026-11-08
- Will Qwen4 Plus be open weights? Reddit parses Alibaba's release clues — Axiomexx · 2026-11-08
- OpenAI exec floats $50/month ChatGPT Pro tier between Plus and $100 Pro — haider1 · 2026-11-08