Autoencoder experiments show bias-free activations yield monotonic representation evolution
ZimingLiu11 · x · 2026-10-31
Follow-up in Ziming Liu's training-tricks thread: in an autoencoder setup, the unshifted (bias-carrying) model recovers recognizable image content from its activation-bias component early in training but relies on it less later, while the shifted model shows clearly monotonic representation evolution with little content in that component.
Consistent with the previous finding: shifting activations down (ReLU−0.2 etc.) not only lowers late-stage loss but makes representation learning trajectories smoother and more predictable.
More from Research
- Overfitting explained with two students and two exams — cneuralnetwork · 2026-10-31
- Betelgeuse runs 68GB Qwen MoE bit-exact at 22 tok/s on RTX 3060 + 16GB RAM — zyxciss · 2026-10-31
- EdgeRock-4B open-sourced with fixed 72MiB KV cache, 99.1% less memory than base — testingcatalog · 2026-10-31
- Claude's Muon optimizer theorem checks out numerically to 4 decimal places — vxnuaj · 2026-10-31
- Dev asks for a mathematician: Claude's Muon optimizer theory is too deep to verify — vxnuaj · 2026-10-31
- Zero-mean 'bug' in ReLU/SiLU/GeLU found to cause loss spikes; fixing it boosts LLMs and ViTs — ZimingLiu11 · 2026-10-31