Qwen 27B local RAG on a 3090 ties Opus 5.5 on Korean benchmark

Front-University4363 · reddit · 2026-10-24

A hobbyist systematically benchmarked local models on a single RTX 3090 using Allganize's Korean RAG set (300 questions, 64 PDFs, known answers and pages):

qwen3.8 27B with thinking hit 77.3% vs Opus 5.5's 76.6% (McNemar p=0.82, a tie); Opus-as-judge gave 80.6 vs 80.3. Disabling thinking drops qwen 6 points.
The model is no longer the bottleneck: of Opus's 70 misses, 22 were scanned pages with no text layer, 23 were retriever failures, only 25 were actual model errors. A 2024 commercial system (Alli + Claude 3.5 Sonnet) only wins on scanned pages (19:1) — the author does no OCR.
Retrieval details matter hugely: BM25 with whitespace splitting got 56% hit@5; a proper Korean morphological tokenizer lifted it to 80%.
gpt-oss 20B scored 57.5% until the author found a 1024-token limit was truncating its thinking; raised to 8192, it hit 69.6%.
KRAFTON's new Raon-1 mini preview (30B MoE, 4B active) lands at 74-75% with 96 tok/s on the 3090, but loops infinitely on 16/320 questions at temp 0 with thinking on.

Caveats: single temp-0 runs, 11 verdicts flipped across hardware, so 1-point gaps are noise; the 20 handwritten unanswerable questions were too easy. Scripts, per-question verdicts and full tables are open-sourced.

Original post →

Qwen 27B local RAG on a 3090 ties Opus 5.5 on Korean benchmark

More from coding & agent