Ox Alpha unmasked: reproducible fingerprint experiments and full logs open-sourced
PawelHuryn · x · 2026-08-25
Pawel Huryn shared the data and reproduction repo behind his Ox Alpha test (phuryn/experiments).- Results: OpenRouter's free stealth model stealth/ox-alpha (GLM family) found only 16 of 105 planted bugs across two repos vs 27 for Grok 4.6, deflating the hype.- Fingerprinting: the stealth-ox-alpha-fingerprint experiment identifies the model from the wire, no benchmark needed — a fixed passage costs 172 prompt tokens on ox-alpha.- Reproducibility: the repo ships exact scripts, unedited logs, and READMEs with method, sample size, and caveats, plus many other model experiments (bug-hunt-bench, fugu-ultra-vs-frontier, kimi-k3-day-one).
Related event: Ox Alpha Benchmarks Mediocre on Coding, Falls Short of Grok 4.6(2 posts)→
More from Models
- AI writing latency stagnant; RL focus may degrade quality — peterwildeford · 2026-08-25
- Before upgrading to a bigger model, test thinking mode: 12 finance prompts compared — KhuyenTran16 · 2026-08-25
- User Shares a Gallery of Gemini Hallucination Examples — Legitimate-Rip-169 · 2026-08-25
- SenseNova U1.5 quantized to run on 12GB VRAM with INT8/W4A8 — junklont · 2026-08-25
- AI struggles with precise terminology in technical writing — Ben_Reinhardt · 2026-08-25
- Startup Accelerated Understanding launches on neural operators, rumored source of huge-context model — inductionheads · 2026-08-25