Experimenting with Nanbeige-Style Triple Loop on Qwen3.5-9B
Important-Farmer-846 · reddit · 2026-08-23
Inspired by the Nanbeige 4.2/4.5 architecture, a developer implemented a "triple loop" structure in Qwen3.5-9B to explore if a model can improve its representation by looping over itself.
Methodology:
- Initial attempts with DeltaNet in middle layers caused forgetting/hallucination.
- Switched to full softmax attention and fixed the LR schedule (dedicated lower LR for the loop), enabling the loop to participate actively.
- Trained using free credits on Modal via distillation (Teacher: Qwen2.5-72B) on a reasoning dataset.
- Training stopped early at 15M tokens due to budget limits (LR decay not applied).
Results:
- Gains: Math (+20%), Long-context (+14%), Instruction-following (+20%), Robustness to paraphrasing (+62%).
- Losses: Reasoning (-10%), Translation (-15%), Coding (-2%).
- The author attributes the reasoning drop to unfinished training (missing LR decay) causing specific failure modes, not an architectural ceiling.
The model is a proof-of-concept, demonstrating the potential of loop architectures when trained correctly.
More from Research
- PROFusion: Robust Real-time Dense Reconstruction via Camera Pose Regression — rsasaki0109 · 2026-08-23
- How to Build Better AI Evals with Claude Code in 5 Steps — petergyang · 2026-08-23
- Graph Convolutional Network by Hand: 12-Step Walkthrough — ProfTomYeh · 2026-08-23
- Napster's Pivot to AI Agents Exposes the Stale Training Data Problem in LLMs — Tanmay_Vermaa · 2026-08-23
- Best-of-N Sampling Test: Math Accuracy Boosted to 65% — SergioPaniego · 2026-08-23
- OpticalDNA: Rethinking Genomic Modeling via 2D Visual Representation — jiqizhixin · 2026-08-23