DeepSeek Distillation Experiment on Gemma 4
Paramecium_caudatum_ · reddit · 2026-07-09
The author reconstructed a dataset using DeepSeek-generated answers, then performed QLoRA distillation training on both Gemma 4 26B and 12B respectively. The post compares their VRAM usage, training loss, and validation performance.
More from Research
- Structural ensembles beat single predictions in TCR:pMHC generalization study — quaidmorris · 2026-07-22
- Structural ensembles, not single predictions, drive robust TCR:pMHC generalization — quaidmorris · 2026-07-22
- A 3D ray plot shows how hard this Jacobian counterexample is to read — moultano · 2026-07-22
- LLM leaderboards are now often measuring the harness too, Gary Marcus warns — GaryMarcus · 2026-07-22
- New paper defines self-state attacks, showing OS defenses leave four agent-memory cases indistinguishable — Justgototheeffinmoon · 2026-07-22
- Krea 2 users recommend a two-pass Clownshark sampler setup for sharper image details — listopalafoto · 2026-07-22