MetaLint: Easy-to-Hard Generalization for Code Linting
Atharva Naik, Lawanya Baghel, Dhakshin Govindarajan, Darsh Agrawal, Yiqing Xie, Daniel Fried, Carolyn Rose
cs.SE, cs.CL, cs.LG
2025-07-16
MetaLint trains linting as instruction following on Ruff synthetic data. Qwen3-4B PEP detection F-score jumps 25.9% to 70.4%, with 26.7% localization F-score matching o3-mini.
Most LLM linters bake a fixed rule set into the weights and treat linting as classification. Rules move with language versions, library APIs, and security guidance, so those models over-flag stale patterns and miss new ones. Prior work already saw this overfitting to a static set.
Difficulty is uneven. Ruff rules such as S104 through S108 (likely hardcoded secrets) are surface cues. PEP 506 asks for secrets.choice rather than random.choice in security-sensitive code, and that depends on intent: not every random.choice is unsafe. Rule engines do not cover that class of checks. Public evals are either linter-generated or unreleased.
MetaLint, from CMU, treats each best practice as a natural-language task. The prompt carries a spec plus examples; the model flags only that idiom and emits JSON line hits, or NO VIOLATIONS FOUND. Flagging any other idiom is penalized. Specs live in the input, so a new rule at test time is a new instruction, not a retrain.
Supervision comes from existing linters. For Python, Ruff (50 of 800+ rules) run on The Stack yields about 53k examples, roughly 1:3 violation to clean. For Java, PMD gives 96.8k examples over 269 idioms, and tree-sitter queries over 8 JEPs give 127.3k. Specs are scraped from official docs.
Two-stage training:
SFT tends to memorize training rules. DPO is what pushes consistent localization under a given spec, and that is where transfer shows up. A CoT variant uses the same pipeline, keeping only traces whose parsed answer is fully correct (reward 1).
The hard eval is a human PEP benchmark: 15 rules outside Ruff, mined from Stack v2 with high-recall heuristics then hand-checked. 536 items, about 52/48 violation vs clean, Cohen's κ=0.95. Train and test files are hash-deduplicated.
On held-out Ruff idioms, Qwen3-4B without CoT goes from 35.4% detection F-score to 45.0% after SFT and 78.7% after DPO; localization F-score goes from 14.4% to 66.1%. SFT's detection F-score gain is almost entirely in-domain (+61.1 points). Far transfer barely moves (+1.4). DPO then adds 40.3 points on far transfer, to 77.9%. The CoT base is already strong, so DPO's extra is smaller.
The same pattern holds in Java. Llama-3.2-3B on PMD in-domain goes from 1.3% detection F-score to 58.9% after DPO. Trained on PMD and tested on unseen JEP idioms, it reaches 61.8% detection F-score and 37.2% localization F-score.
The hard PEP set is the main test. Qwen3-4B:
| Setup | Det. F | Det. recall | Loc. F |
| Qwen3-4B base | 25.9% | 17.2% | 8.8% |
| +SFT | 13.8% | 8.2% | 2.9% |
| +SFT+RS-DPO | 70.4% | 70.4% | 25.0% |
| CoT base | 53.5% | 39.9% | 18.8% |
| CoT+RS-DPO | 64.7% | 49.6% | 26.7% |
SFT regresses on hard PEPs; it is memorizing Ruff. Skipping SFT and running DPO from the base model collapses the output format and scores near zero. Training on the same 50k Ruff files with a generic "find code quality issues" prompt, then scoring by PEP kind, yields 0.00% detection and localization F-score. The spec is the transfer mechanism. Extra data without it does not transfer.
Against larger models, the 4B non-CoT run has the highest detection recall at 70.4%. Detection F-score 70.4% sits next to o3-mini at 70.7%, below GPT-4o at 77.2% and GPT-4.1 at 75.5%. CoT localization F-score 26.7% matches o3-mini at 27.1% and sits well below GPT-4.1 at 46.5%. False positives are the cost: 24.5% FPR on 257 clean files for non-CoT, 5.1% for CoT. GPT-4o sits at 9.7% FPR with 67.9% recall.
On the 536-item PEP set, non-CoT MetaLint is about 175 times cheaper than GPT-5. On Flask (83 files times 15 PEPs), the CoT 4B model is 9.5 to 21 times cheaper than Claude Sonnet 4.6 / Opus 4.7 via Claude Code. Cost scales with file-rule pairs, not conversation context.
The useful slot is a cheap critic inside an agent pipeline. Linting needs little repo context and no execution. A 4B model fits on one GPU; a new rule is a new natural-language spec.
This is a clean easy-to-hard result, not a generic linter finetune. Localization is still weak, and high recall without CoT comes with 24.5% FPR. Fine as a scanner with a human in the loop. Too early as the only gate.
The appendix is frank: PEP text may have been in pretraining (base-model controls isolate the training gain, not contamination); they did not distill CoT into a non-CoT model; self-generated CoT tended to copy hints in the prompt; only DPO, no PPO or GRPO; no curriculum over abstraction; Python and Java trained separately; detection only, no patches; per-idiom prompts, no batched multi-rule serving.
A few holes remain. The hard set is 15 PEPs and 536 items, a narrow distribution for a generalization story. FPR already shows that high recall is expensive; they treat FPR as a knob via the clean-example fraction in DPO, but they do not publish a full precision-recall curve at a production threshold. Java "unseen" is PMD and JEP swapping roles, and those specs differ in style; JEP to PMD is weaker than the reverse, so the safer reading is diversity of training specs, not deep semantic transfer. Localization F-score stuck at 25% to 27% is not yet a linter.