Dual-mask fine-tuning recovers cacheability with only -0.4 DI vs full bidirectional
antoine_chaffin · x · 2026-10-17
To fix the cache-reuse problem caused by fully bidirectional attention, the author benchmarked several attention schemas:
- Naive PrefixLM-style (bidir state, causal question): 59.53, cacheable but -2.0 vs full bidir; half the gap comes just from making the question causal
- Both bidir but state can't see question: 60.86 cached, only 61.09 even uncached — still a regression
- Winning recipe: train fully bidir, then a small fine-tune removing state->question attention. Reaches 62.16 uncached (same or better than v1.1) and 61.18 cached (only -0.4)
You can push the tradeoff further by caching only a few bottom layers, MICE-style. The attached chart shows the full design-space table.
More from Models
- Claude cracks 30-year mystery: ECDSA curve seeds were just RNG output — scaling01 · 2026-10-17
- KOL teases mystery model argon, says people are 'severely underestimating' it — iruletheworldmo · 2026-10-17
- Reddit post argues Anthropic has led benchmarks for 6+ months and is 2-3 iterations ahead internally — Concern-Excellent · 2026-10-17
- Perplexity traces decider v1.2 regressions to longer training, mitigates via model souping — antoine_chaffin · 2026-10-17
- Perplexity open-sources next-gen decider models in 27B and 4B sizes with up to 5x faster inference — antoine_chaffin · 2026-10-17
- Perplexity ships pplx-decider v1.1: bidirectional attention adds +2.8 DI, new sizes 27B/4B — antoine_chaffin · 2026-10-17