Two DGX Sparks run a 321B model with 819K-token context, cache survives restarts
AIFlow_ML · x · 2026-11-22
@greatYiTin runs Sage-S2.1-Flash on a pair of DGX Spark units at up to 37 tok/s with 0.16 s time-to-first-token and an 819K-token context window.
The standout detail: the prompt cache survives restarts. In long-context workloads, rebuilding a huge cache after every crash is where things get painful, so persistent caching makes local long-context inference far more practical.
More from Infra
- Inkling 2 Mini on 2× RTX 3090: 557K context + vision at ~100 tok/s with NInfer-all — Sisuuu · 2026-11-22
- John Langford weighs in on FBT note: fixed-point convergence and quantized caching — JohnCLangford · 2026-11-22
- Fewer than 10 companies can safely move tenants across multi-tenant GPU clusters, argues compute-market skeptic — AccBalanced · 2026-11-22
- Tens of thousands of compute nodes power this multi-agent swarm architecture — hwchase17 · 2026-11-22
- Teknium shares Hermes telemetry: 92.4% of tokens are cache reads — Teknium · 2026-11-22
- Musk demos real-time AI video generation on half of one AI4 TRIP accelerator — jamesdouma · 2026-11-22