Gemma translation tests exposed a boundary failure between instructions and payloads
ReinforcedKnowledge · reddit · 2026-07-22
A translation model can start solving instead of translating
The author describes a failure mode encountered while translating samples with Gemma models: instead of translating the text, the model sometimes executed the instructions embedded in the payload. In practice, reasoning traces led it to solve programming problems, proofs, and other tasks rather than simply translate them.
The core lesson
The write-up argues that the boundary between instruction and payload broke down. The same risk may extend beyond translation to rewriting, proofreading, and summarization whenever an instruction-following model is asked to transform text that itself contains instructions.
What helped
The author notes that the issue looked fine from the outside because jobs completed normally, but quality was broken. A cheap alarm based on output-to-source length ratio helped surface the worst failures. Promising mitigations included:
- chunking the input
- protecting code fences and mathematical structure
- building more rigorous input tagging
The post is explicit that the observations are specific to RedHatAI/gemma-3-27b-it-FP8-dynamic and RedHatAI/gemma-4-31B-it-FP8-Dynamic, and that broader conclusions would need more testing.
More from Research
- New NBER paper on how organizations use AI completes a three-paper series — daveholtz · 2026-07-22
- OAT uses 100 successful trajectories to debug failing AI agents without failure labels — TheTuringPost · 2026-07-22
- MoE, the mixture-of-experts architecture behind many top LLMs — vista8 · 2026-07-22
- NexForge synthesizes agent training data from requirements and lifts Qwen3.5 by 30 points — nex-agi · 2026-07-22
- SeerGuard uses a world model to screen risky actions in mobile GUI agents — Xue Yu · 2026-07-22
- A 15-hour video fine-tune turns masked trajectories into a robot control interface — Hadi Alzayer · 2026-07-22