MameLoshnLM: Yiddish Language Model and Evaluation Benchmark
Uri Katz, Omer Goldman, Tomasz Limisiewicz, Reut Tsarfaty, Noah A. Smith
COLM) 2026
cs.CL, cs.AI
2026-08-06
Continued pretraining of Llama 3.1 8B gives the first open-source 8B Yiddish LM, averaging 62.6 on a new 9-task benchmark; an audit shows only 42% of mC4 Yiddish is genuine.
Yiddish is a Germanic language written in Hebrew script, spoken by about a million people worldwide, with a thick layer of Hebrew/Aramaic-derived vocabulary (loshn-koydesh). It has a vast literary tradition but a tiny digital footprint, and the Yiddish text sitting inside web-scale multilingual corpora is mostly not the real thing.
The authors audit the Yiddish split of mC4, the multilingual corpus many models train on, and the result is direct: only 42.2% of documents are genuine Yiddish from validated native sources. Another 29.8% is machine-translated, 21.9% is Hebrew that a language identifier misclassified as Yiddish, and 6.2% is miscellaneous fragments. The "Yiddish" that multilingual models actually learn from is, in the majority, noise.
The measurable consequence is that multilingual models trained on this data produce Yiddish that sounds fluent while dropping the features that make the language itself. Before this work there was no dedicated Yiddish language model and no reliable evaluation benchmark for one.
Three artifacts ship together: the Oytser corpus, the Kashes benchmark, and the MameLoshnLM model.
Oytser (אוצר, "treasure") is a high-quality Yiddish pretraining corpus: 915M words, 346.3K documents, 5.28B tokens, from nine sources. The largest by far is the Yiddish Book Center digital library, 12.3K books and about 723M words, OCR'd with Jochre3, mostly from the past century, spanning literature, essays, and history. This is what pulls Yiddish's long literary tradition into the training data. The other half is contemporary web-native text: forums (Ivelt, Kaveshtiebel, written in a freer, more dialectal style), news and magazines (Forward, the Yiddish Wikipedia, yiddish.news, others), plus a Yiddish translation of the Hebrew Bible. PII was stripped with regex.
Kashes (קאַשעס, "difficult questions") is the accompanying benchmark, nine tasks across translation (a new Kashes-mt set of 5,287 pairs plus FLORES+), linguistic analysis (POS tagging, dependency parsing, lemmatization, transliteration), named entity recognition (EHRI-NER, WikiANN, newNLP), and language understanding (PIQA, WikiQA, PAWS-Wiki). EHRI-NER and newNLP were built by digital humanities scholars to process Holocaust testimonies and historical newspapers; the benchmark reuses them.
For the model, the team continues pretraining Llama 3.1 8B: 5.7B tokens, 36,663 steps, about 207 GPU-hours on a single H200. The data mix is 72% Yiddish by words (92% of tokens) and 28% English. That mix is deliberate. The authors tried reallocating part of the English budget to historically related languages, German, Hebrew, Polish, and Russian, and it did not help; on several Yiddish-centered tasks those variants underperformed even the un-adapted base Llama. A strongly Yiddish-dominant signal plus a little English beats pulling related languages in.
5-shot averages across 14 evaluations:
| Model | Average |
| MameLoshnLM | 62.6 |
| Gemma-2 9B | 57.0 |
| Llama 3.1 8B | 56.8 |
| Qwen3 8B | 54.7 |
| EuroLLM 9B | 45.6 |
| BLOOMZ 7B | 20.9 |
Qwen3 is the only baseline whose documentation explicitly lists Yiddish support. The gains land on Yiddish-centered tasks: English-to-Yiddish FLORES+ COMET is 78.5, more than 11 points above the closest competitor; EHRI-NER micro-F1 is 41.3 against Llama's 34.2. The model stays competitive on general tasks like WikiANN and WikiQA. The two it loses, PAWS-Wiki and PIQA, are machine-translated versions from the Aya benchmarks.
More revealing than the leaderboard is the linguistic probe. Much of what makes Yiddish Yiddish is the loshn-koydesh (LK) Hebrew/Aramaic lexical layer. The authors test how well each model produces LK words:
| Probe | MameLoshnLM | Llama 3.1 8B | Gold |
| LK content-word rate | 4.7% | 1.6% | 6.2% |
| LK sentence-match rate | 52.4% | 16.0% | n/a |
| ge- participle accuracy | 50.8 | 5.1 | n/a |
| Hebrew-origin plural accuracy | 30.2 | 2.3 | n/a |
The little LK that Llama does produce is concentrated in proper nouns (Israel, Rebbe, Torah), not everyday vocabulary. This tracks its training data: machine-translated mC4 pages have an LK rate of 3.6% against 10.2% for validated native sources, and Llama's output falls below even the machine-translated text. The model reproduces the lexical depletion it was trained on.
For anyone building models for low-resource languages, this is a directly copyable recipe: curate a corpus, build a targeted benchmark, continue-pretrain an existing model. A single H200 and 207 GPU-hours produced a language-specialized model at low cost, and both Oytser and Kashes are open.
For multilingual NLP more broadly, the contribution is not the model but the audit and the probes. They give quantifiable evidence that web-scale multilingual corpora are bad for low-resource languages in a specific way: they support surface fluency while eroding the features that define the language. That almost certainly extends beyond Yiddish to any language whose web footprint diverges from real usage, including diglossic languages and languages with a fragmented online presence.
Be clear about what this is: a domain-adaptation and data-engineering result, not a new architecture. Its value is in the data work and the linguistic analysis, not in modeling novelty.
The model recovers only part of the LK gap: 4.7% against a native 6.2%, still short of gold.
The license is non-commercial. Institutional agreements with the Yiddish Book Center and In geveb lock MameLoshnLM to a non-commercial license, so building products on it is off the table.
There is no instruction tuning. This is a continued-pretraining base model, not a chat model; the authors list instruction tuning as future work.
The translation metric has a blind spot. The authors themselves show COMET is largely insensitive to LK loss (per-sentence LK recall correlates with COMET at only ρ=0.23). Any translation gain carried by COMET could hide lexical erosion; MameLoshnLM covers this with its own LK probe, but other models rarely get checked this way.
The "related-language mixing doesn't help" finding comes from a single 5.7B-token run. Whether it holds at larger scale is not tested.