Fleuret and giffmana debate numerical blow-up risks in an activation-norm formula

giffmana · x · 2026-11-08

François Fleuret floated an activation-normalization variant y = x / log(exp(‖x‖) − 1 + exp(0.1)), uneasy that rare numerical paths could bite over millions of training steps. giffmana (lucidrains) countered that under the stated assumption it's just an expensive identity and its log-exp structure explodes exponentially fast, suggesting plugging in common activation norms to verify.

Related event: François Fleuret demonstrates RMSNorm gradient blow-up and community explains why training rarely hits it(9 posts)→

Original post →

More from Research

• DECIDE benchmark tests voice-agent decisions on 4,867 questions from 1,081 real calls, scoring 20 models — Aiden_Tech_Ai · 2026-11-08
• YC amplifies demo of models handling semi-serious X-band phased-array design — ycombinator · 2026-11-08
• Analysis of 250k ICLR reviews finds 65% AI-written, scores collapsing to middle — TuhinChakr · 2026-11-08
• Rewritten DeepSWE trips GLM and DeepSeek, pointing to evaluation-aware benchmarks — teortaxesTex · 2026-11-08
• Israeli researchers report method letting AI teach itself without human data — windless12 · 2026-11-08
• HARDEN makes benchmarks harder without changing answers to test real robustness — burkov · 2026-11-08

Research channel →