Fine-tuning Gemma 4 26B on 500k court decisions never beats the prompted base model
seruZ12 · reddit · 2026-08-20
A Reddit user spent days fine-tuning Gemma 4 26B A4B (both base and IT variants) to generate paragraph-level legal principles/headnotes from court decisions, using the highest-quality subset (20% with rationales) of a 500k-decision database, with an unsloth pipeline on a rented server and Claude vibe-coded tooling.
Neither fine-tune beat the simply-prompted base model on their LLM eval. The only apparent win came from an eval built around a specific passage-extraction task, but the outputs turned out unusable — full of padding and generalizations. They're asking the community whether a 26B model can do this at all, and what gotchas to watch for.
More from Research
- SkillForge: Self-Distilling Agents for Project-Specific Bug Fixing — SJTU · 2026-08-20
- FM-Bench Evaluates Long-Horizon Agent Management via Football Club Simulation — Tianyou Wang · 2026-08-20
- Research: Looped Language Models Improve Compositional Tool Calling — Andrei Cristian Popescu · 2026-08-20
- Paper proposes interpretable framework for dynamic hedging using Itô Signatures — chaumian · 2026-08-20
- Paper Proposes Multi-Level Market Making Using Reinforcement Learning — chaumian · 2026-08-20
- Discussing the feasibility of representing ontologies in plain Markdown — durlabha · 2026-08-20