Score Centering Is Infra-Unfriendly at Long Context: Two Huge [T, K] Tensors and a Custom Kernel
dejavucoder · x · 2026-11-20
dejavucoder quote-tweeted a technical remark noting that score centering is not very infra-friendly under long context: it requires storing two huge [T, K] tensors and doing element-wise multiplication, plus a custom kernel. A short engineering note on attention/scoring implementation details.
More from Infra
- 753B-parameter GLM-5.3 served fully in GPU memory across two NVIDIA DGX Stations — kuchaev · 2026-11-20
- PolyAI's real-time dialog model Dialog-RSN-1 lands on SageMaker JumpStart — AWS ML Blog · 2026-11-20
- Three decommissioned V100s in a $1,042 rig beat a $5,399 Mac Studio at long contexts — bittingthembits · 2026-11-20
- NVIDIA's Rubin debuts in MLPerf Training and Inference as benchmark maintainer congratulates strong quarter — TheKanter · 2026-11-20
- Halocline: hand-rolled CUDA engine runs Qwen4 on one RTX 4090, up to 2.6x faster than Strata — Distinct-Pie2389 · 2026-11-20
- US chip curbs bite: Moonshot drops $150M Indonesia deal, TSMC and Samsung halt China shipments — The Decoder · 2026-11-20