NOLLI: A Difficulty-Calibrated Puzzle Benchmark for Diagnosing the English-Korean Performance Gap
Dasol Choi, Joonyong Park, Daegon Yu, Soo Yong Kim, Youngsook Song, Seunghyeok Hong
cs.CL
2026-08-05
A procedurally generated EN-KO puzzle benchmark (15 types, 7,500 items) on 15 models: matched translations are equivalent within plus or minus 10pp, but Korean Cipher trails English by up to 68.7pp, pinpointing multi-step sub-syllabic execution.
When a large model does worse in Korean than in English, what exactly is failing? Existing multilingual benchmarks usually hand back a single blurry "Korean is X points lower than English", conflating the language itself, the writing system, and Korean-specific knowledge. Static benchmarks are also easy to contaminate and saturate, and their difficulty is often defined by task structure (grid size, chain length) rather than by what models actually find hard.
NOLLI aims to diagnose, not to rank. With procedurally generated English-Korean puzzles it separates three effects, presentation language, writing system, and Korean-specific knowledge, so each can be inspected alone.
NOLLI has 15 puzzle types, 25 tasks, and 7,500 items. Every item is produced by a task-specific generator from a random seed, so it is regenerable, verified to have a unique solution, and scored deterministically with no LLM judge anywhere in the loop.
The crux is difficulty calibration. Rather than equating harder with bigger, the authors calibrate behaviorally: for each puzzle type they tune the generator parameters until a fixed reference model (Gemini 3 Flash at medium reasoning effort) lands in a target accuracy band, Easy 75±10%, Medium 50±10%, Hard 25±10%. The three tiers are thus defined by "what score the model actually gets", not by intuition. Of 75 task-tier combinations, 72 land in band.
The three-level cross-lingual design is what makes diagnosis possible. The first level is direct translations (8 puzzle types, same generator and parameters in both languages, hand-written templates, no machine translation), which isolates the cost of presentation language alone. The second level is script adaptations (Cipher and Cryptarithmetic), present in both languages but with the Korean variant rebuilt around the compositional structure of Hangul jamo, the sub-syllabic letters inside a Korean syllable block. The third level is Korean-only (5 types: Kinship, Saju, Time, Korean Units, Jamo Composition), with no English counterpart.
Jamo is the key idea: a Korean syllable block decomposes into an initial consonant, a medial vowel, and optionally a final consonant. The Korean Cipher operates on jamo indices; Jamo Composition decomposes and recomposes syllables.
The authors evaluate 15 models, including the frontier GPT-5.5, Claude Opus 4.8, and Gemini 3.1, open-weight Llama, Qwen, Gemma, and DeepSeek, and Korean-developed EXAONE, Mi:dm, and Solar. Three that score below 3% overall are excluded as too weak; the gap analysis runs on the remaining 12.
The first level gives a clean result: on matched direct translations, the English-Korean accuracy gap is only −2.7 to +5.0 percentage points, statistically equivalent within a ±10pp tolerance (TOST). Simply presenting the item in Korean barely costs anything.
The gap concentrates in the writing-system level. Korean Cipher trails English by up to 68.7 percentage points: DeepSeek-V4-Flash scores 74.3% in English against 5.7% in Korean. Every non-frontier model with nonzero English accuracy loses at least 21 points; Qwen3.5-27B, EXAONE-4.0, Solar-100B, and gpt-oss-120b sit at 21% to 56% in English but below 1% in Korean.
The contrast on the next task over is sharp. Over the same jamo, Cryptarithmetic (treating jamo as opaque symbols in a letter-arithmetic puzzle) shows no systematic Korean penalty, and 6 of 12 models actually score higher in Korean. Cipher requires decomposing a syllable, doing index arithmetic, and recomposing; Cryptarithmetic only maps jamo to digits. So the failure is not "the tokenizer cannot handle jamo", it is "multi-step chaining over jamo".
| Task type | Operation | Korean result |
| Direct translation (8 types) | same item, other language | equivalent to English (±10pp) |
| Cryptarithmetic | jamo as opaque symbols | no systematic penalty |
| Korean Cipher | decompose, index arithmetic, recompose | trails English by up to 68.7pp |
Two pieces of evidence close the loop. First, Jamo Composition accuracy predicts Korean Cipher accuracy with R²=0.91, and a partial correlation of 0.87 survives after controlling for direct-translation Korean score. Second, in the error analysis, strong models recover most jamo even when wrong (similarity 0.85+), weak models stay at baseline (0.11 to 0.21), and clean single-step slips such as "shift a correct jamo stream by one position" almost never happen, so what breaks is the multi-step chain, not a single step.
The Korean-only Kinship task exposes a different deficit: positive in all 12 models, and it does not shrink as models get stronger overall (r=0.16). Inside the frontier group the ordering even inverts, Gemini 3.1 (69.0) above Opus 4.8 (63.3) above GPT-5.5 (45.7); the strongest model overall, GPT-5.5, has the largest Kinship deficit (−36.7pp).
A final methodological warning: a salient "size" measure fails to grow from Easy to Hard in 7 of the 15 types. In Kinship the chain length actually shrinks (4 to 5 hops Easy, 2 to 3 hops Hard) yet reference accuracy falls from 77% to 31%. Using structural size as a proxy for difficulty is unreliable.
For benchmark and multilingual-model work, NOLLI is a reusable, contamination-resistant diagnostic that splits a single "Korean is worse" into three locatable things. The most actionable conclusion is that the Korean gap is mostly not a tokenizer hitting jamo, it is the model's inability to do multi-step operations over sub-syllabic letters. That means re-tuning the tokenizer for finer jamo may not be enough; models need training that actually exercises multi-step letter-level operations.
The behavioral difficulty calibration is also worth borrowing: rather than assuming bigger means harder, let a reference model tell you which generator parameters correspond to which difficulty tier.
The authors are explicit that these contrasts are diagnostic, not causal. The script adaptations and Korean-only tasks differ from direct translations in task content and procedural demands, so they localize candidate bottlenecks but cannot prove causation.
For the sub-syllabic claim specifically, the correlation covers only 12 models, and both Jamo Composition and Cipher operate on jamo. Without measuring per-model jamo merging or adding an English character-level control, one cannot separate a Korean-specific bottleneck from a general weakness in subword manipulation. The Korean-only tasks differ from the direct-translation baseline in task family, answer space, and context length, and Kinship is the only cultural-lexicon task, so generalization to broader Korean cultural knowledge is limited.
Difficulty calibration targets one reference model at one reasoning effort, so absolute band membership need not transfer to other models; a held-out check on six types preserves ordering in all cells and band membership in most, but full-suite stability is unverified. And equivalence is relative to a ±10pp margin, not a claim of equal accuracy.