Gisting compresses prompts by 75% to cut LLM costs while preserving behavior
morgymcg · x · 2026-08-22
The post introduces "Gisting," a technique that finds a set of tokens eliciting the same behavior as a full System Prompt. This method can reduce prompt size to 25%, significantly increasing throughput and lowering costs.
Shopify Engineering Implementation:
- Compression: Shopify compressed its GraphQL agent's system prompt from 6,000 to 1,500 gist tokens (4:1 ratio) without losing prediction quality.
- Performance Gains: At 350 RPM, median TTFT dropped from 438ms to 354ms, end-to-end latency from 6.8s to 4.2s, and QPS rose from 20.2 to 23.4.
- Mechanism: Special token embeddings are learned via knowledge distillation and swapped in at inference time to mimic the full prompt while model weights remain frozen.
- Resource Savings: The increased throughput allowed for a reduction in allocated GPUs for the agent's traffic.
Related event: Gisting Compresses Prompts to Cut LLM Costs by 75%(3 posts)→
More from Infra
- PyTorch now runs natively on AWS Trainium with zero code changes — PyTorch · 2026-10-20
- NeurIPS 2026 paper 'Intelligence per Watt' measures on-device AI efficiency across 1M queries — HazyResearch · 2026-10-20
- SpaceX Faces $26B Compute-Driven Earnings and a 1.3B Share Unlock in the Same Week — JOBhakdi · 2026-10-20
- Core economic flaw in the AI bull case: compute cost exceeds the benefit it delivers — SumitGup · 2026-10-20
- ijah: a new engine for running Qwen 3.8 Flash-Next locally on M5 Max Macs — SeveralViolins · 2026-10-20
- Shadeform launches six features as agent-era compute breaks existing rental models — omarsar0 · 2026-10-20