Teaching Nemotron Greek: Mining a Corpus, Adapting Retrieval, and Grounding Generation for Modern Greek across Specialist Domains
Ayoub Kirouane, Christos Petrocheilos
eess.AS, cs.AI, cs.CL
2026-08-06
A parameter-free BM25 beats every multilingual embedder on specialist Greek text, an 8B included. Fine-tuning a 1B lifts nDCG@10 from 0.362 to 0.835, then loses to BM25 off-domain.
Modern Greek is missing from NVIDIA's Nemotron retrieval models and from every major multilingual retrieval benchmark: BEIR is English-only, MIRACL excludes Greek. That means the embedders were neither trained nor measured on Greek. Yet Greek legal, energy, financial, and clinical documents are exactly the long, jargon-dense text that retrieval-augmented generation is meant to serve.
Production RAG stacks follow a common practice: ship a multilingual embedder and drop BM25. This paper asks what happens to a language that is out of distribution for both the models and the yardsticks.
The authors adapt the Nemotron retrieval stack end to end for Modern Greek, across energy, legal, finance, and medical domains plus a general foundation slice. Several engineering decisions matter.
Building the corpus turned out to be half the work. From 407,053 raw (query, document) pairs, 65,773 clean records survive. Two findings were forced out by the data: there is essentially no commercially licensed native Greek instruction data at scale, so anything useful has to be translated; and Greek document chunks are long (positives median 1,196 characters, about 1,034 tokens), so the maxlen of 512 that most retrieval tooling defaults to silently truncates 87% of training pairs. Both models are trained at 4,096.
Queries are synthetic, but how they were synthesized matters more than that they were. A bare instruct model writes short keyword queries that look nothing like real users' long, formal, fact-seeking questions aimed at a specific decision number or date. Few-shot style-matching against real production queries doubled the reranking lift over BM25, from +0.040 to +0.081 nDCG@10.
Negatives are the quality lever. A reranker learns only from contrast, so negative quality sets the gradient. They use Qwen3-Embedding-8B to mine a cosine-kNN window per query inside its own domain (skipping the most similar positives, dropping near-duplicates), keeping about six negatives per query. Mining negatives with BM25 instead was measurably worse: the lexical signal that makes BM25 a strong retriever does not make it a good negative miner.
The retriever is a full fine-tune of Nemotron-3-Embed-1B with InfoNCE. The reranker is a pointwise cross-encoder. The reader is a LoRA on Nemotron-3-Nano-30B-A3B, a 3B-active MoE, with only 439M trainable parameters (1.37%). For evaluation they built HERA, a 4,946-item Greek Wikipedia long-retrieval benchmark that deliberately includes 25% unanswerable and 23% multi-hop items.
The sharpest result was unexpected: on five specialist domains, a parameter-free BM25 baseline beats every off-the-shelf multilingual embedder, including the 8B one. BM25 scores 0.757 nDCG@10, Qwen3-Embedding-8B scores 0.680, and the untouched Nemotron scores 0.362. Scaling does not rescue it: within the same Qwen3 family, 8B and 4B differ by −0.004, and dropping to 0.6B costs only 0.070; the whole family sits below BM25 across a 13× parameter range. What is missing is not capacity but exposure to the language.
After fine-tuning the 1B embedder, nDCG@10 climbs from 0.362 to 0.835, with every domain gaining at least +0.42.
| Method | nDCG@10 | Note |
| BM25 (parameter-free) | 0.757 | Beats every off-the-shelf embedder |
| Qwen3-Embedding-8B (off-the-shelf) | 0.680 | Scaling does not help |
| Nemotron-1B (unadapted) | 0.362 | Near-nonfunctional on Greek text |
| Nemotron-1B (fine-tuned) | 0.835 | Beats BM25 in-domain |
That win reverses off-domain. On HERA's retrieval track (4,946 queries over 300,000 general Greek Wikipedia passages), the language competence transfers (the fine-tuned model beats its unadapted base by +0.399), but the advantage over BM25 does not: BM25 beats the fine-tuned model by +0.081, and the in-domain ordering flips. The fix the authors land on is pragmatic. Do not replace BM25; fuse it with the dense model by reciprocal rank fusion (RRF). The fusion beats the stronger of the two components by +0.013 in-domain and +0.027 out-of-domain. Run both.
The reranker carries a methodological warning. An off-the-shelf cross-encoder contributed nothing distinguishable from not reranking at all on the first evaluation (−0.006, p = 0.52); only after fine-tuning was it worth +0.047. Without the no-rerank floor as a control, you cannot tell whether a reranker belongs in the pipeline at all.
For reading, after the LoRA on the 30B-A3B model, judged answer correctness rises from 29.4% to 66.9% and faithfulness from 25.2% to 84.5%, a 3.4× gain. Chained end to end with both stages swapped, nDCG@10 goes from 0.559 to 0.848 and Recall@10 from 0.624 to 0.955, meaning the off-the-shelf stack never even put the answer in context for about 38% of queries.
For anyone building RAG, the direct lesson is to measure your lexical baseline before buying a bigger model. On specialist Greek, swapping BM25 for an off-the-shelf multilingual embedder made retrieval worse. Scaling was the wrong axis; language exposure was the right one. On general Greek the ordering reverses again, so do not bet on one side: fusing the two is the robust move.
The methodology models two things. First, evaluate a reranker against the no-rerank floor; comparing reranker A to reranker B cannot tell you whether either belongs. Second, re-run your results: two of the authors' own conclusions did not hold on a second, larger evaluation. That kind of honesty about how instruments lied is rare in the RAG evaluation swamp.
Keep the boundary clear: the advantage is in-domain, for specialist domains. Read it as fine-tuning that fixes specialist Greek, and you will be disappointed.
Every retrieval number is measured on synthetic queries, which echo their source passage's vocabulary and are systematically easier, so the numbers are an upper bound on ranking systems against each other and overstate the production lift. The authors measured real production traffic: the margin over BM25 shrank to about +0.04, under half of the +0.080 on the synthetic set.
The reader numbers are single-run point estimates with no intervals, and the judge model shares a family with the model that authored HERA's items, an LLM-as-judge family-preference confound, so judged correctness is an upper bound.
Abstention, refusing an unanswerable question, only improves from 1.2% to 30.5%, still a failing grade; the fine-tuned reader still confidently answers most unanswerable questions. On the cost side, English MMLU drops 0.098 and arc-challenge 0.094, so forgetting is real. HERA's queries are also LLM-generated and share vocabulary with their gold passages, which flatters BM25's absolute standing on both benchmarks.
There is no zero-replay control arm, so the Greek gains cannot be cleanly attributed to the adaptation rather than the English replay slice; and the retrieval training corpus is not released, so results reproduce against the published models but cannot be retrained from scratch.