Is SwiGLU's Open Positive Tail Necessary? Evidence from Closed-Tail Gating with MemGLU
Yuting Ge, Pengju Yang, Mingkai Nie
cs.LG
2026-08-07
Paired 9M/30M runs swap SwiGLU for MemGLU, a bounded gate that closes on both sides; validation NLL stays within ~0.1% either way, so the open positive tail is not required.
SwiGLU is the default feed-forward design in LLaMA, Qwen and PaLM: the gate gσ(g) multiplies the value path element-wise and controls how much of each channel passes. One geometric property gets inherited along with it: for large positive g, σ(g) approaches 1 and the output grows linearly without bound, an open positive tail. Every model that adopts SwiGLU takes this shape by default, and widespread adoption is not evidence of necessity. The paper narrows the question to one sentence: close that tail completely, and can a language model still be trained to the same loss?
The core is a controlled swap. MemGLU's gate is φ(g) = c₀·tanh(g)·sech(g): odd, bounded, peaking at |g| = arsinh(1) with magnitude 1/2 and decaying to zero on both sides. Since sech(g) = 1/cosh(g) decays exponentially in |g|, this is a band-pass gate, open in the middle and closed at both ends, set against SiLU's unbounded positive pass-through.
Two choices make the comparison clean. Only the scalar gate changes: projections, parameter counts and FFN dimensions stay identical, so any observed difference traces to gate shape. And c₀ is RMS-matched, calibrated so the gate output's root-mean-square at initialization equals SwiGLU's (c₀ ≈ 0.604 at 9M, 0.699 at 30M). Skip that step and initial activation magnitudes differ, leaving shape and scale confounded.
Training is strictly paired: within each pair the two models share initial weights, data order, token budget and optimizer settings (AdamW, lr 3×10⁻⁴), across three seeds. The 9M model (6 layers, dmodel 192) sees 50M tokens; the 30M model (10 layers, dmodel 384) sees about 500M.
The memristor backstory, where the shape falls out of the antisymmetric branch separation of a first-order memristor model, is mostly provenance. The function is a mathematical sibling of ReSech (x·sech(x)) from 2016, which the paper itself notes in related work. The shape properties do the work, not the device physics.
Validation NLL is the primary metric, lower is better:
| Scale | Gate | Validation NLL (3 seeds) | Relative | Paired wins |
| 9M | SwiGLU | 5.7267 ± 0.0077 | reference | – |
| 9M | MemGLU (RMS-matched) | 5.7203 ± 0.0024 | −0.111% | 3/3 |
| 30M | SwiGLU | 3.5540 ± 0.0002 | reference | – |
| 30M | MemGLU (RMS-matched) | 3.5583 ± 0.0038 | +0.122% | 0/3 |
The closed-tail gate wins all three paired seeds at 9M and loses all three at 30M. The sign flips while the magnitude stays near 0.1%. The 0/3 record at 30M means that small deficit is consistent rather than noise, but 0.1% cannot support a claim that the open tail is what makes SwiGLU work.
Interventions supply the other half of the evidence. Trained SwiGLU checkpoints are evaluated with the positive tail capped, smoothly attenuated, or hard-removed:
| Intervention | 9M ΔNLL | 30M ΔNLL |
| Positive cap | +0.032 | +0.122 |
| Smooth attenuation | +0.093 | +0.303 |
| Hard removal | +0.403 | +0.960 |
A trained SwiGLU model does use its tail: the harder the cut, the worse the loss, and the larger model depends on it more, with hard removal costing nearly one NLL at 30M.
Diagnostics reconcile the apparent contradiction. MemGLU models do not avoid the descending region beyond the gate peak; occupancy there even rises at 9M (2.688% to 5.230%, while falling from 5.002% to 3.954% at 30M). But that region's share of activation energy collapses: at 30M, its share of gate-output energy drops from 50.857% to 8.209%, and of the GLU product from 54.921% to 8.573%. The paper calls this occupancy–energy decoupling: the region stays populated while the energy budget moves elsewhere. One more detail: c₀ matched magnitudes only at initialization. After training, MemGLU's gate-output RMS sits at 0.83 (9M) and 0.76 (30M) of SwiGLU's, so optimization itself pushed activations to smaller magnitudes.
The conclusion: near-identical losses, completely different usage. Trained from scratch, optimization organizes FFN usage around whatever gate geometry is available. The gate geometry shapes how the FFN is used, not how well it performs.
Architecture components spread by default inheritance and rarely get isolated tests like this. SwiGLU was adopted wholesale by PaLM, LLaMA and Qwen, and the open tail came along as part of the package. This paper is a clean counterexample: a gate with nearly opposite properties trains to the same loss from scratch.
For anyone doing architecture search, the caution is direct: at tens of millions of parameters, pretraining-loss differences at the 0.1% level with sign flips across scale will not settle an activation-function choice. The paper also separates two questions that usually get conflated: what a model needs during training (flexible, it adapts to the gate it gets) versus what a trained model depends on (very much its positive tail). Both can hold at once.
For the memristor and device-inspired computing community, this is a rare work that pushes device geometry all the way to language-model pretraining, even though the device narrative is not functionally necessary.