Gisting compresses prompts by 75% to cut LLM costs while preserving behavior
morgymcg · x · 2026-08-22
The post introduces "Gisting," a technique that finds a set of tokens eliciting the same behavior as a full System Prompt. This method can reduce prompt size to 25%, significantly increasing throughput and lowering costs.
Shopify Engineering Implementation:
- Compression: Shopify compressed its GraphQL agent's system prompt from 6,000 to 1,500 gist tokens (4:1 ratio) without losing prediction quality.
- Performance Gains: At 350 RPM, median TTFT dropped from 438ms to 354ms, end-to-end latency from 6.8s to 4.2s, and QPS rose from 20.2 to 23.4.
- Mechanism: Special token embeddings are learned via knowledge distillation and swapped in at inference time to mimic the full prompt while model weights remain frozen.
- Resource Savings: The increased throughput allowed for a reduction in allocated GPUs for the agent's traffic.
Related event: Gisting Compresses Prompts to Cut LLM Costs by 75%(3 posts)→
More from Infra
- Alibaba & ByteDance paper: Model inference is no longer the main bottleneck for AI agents — rohanpaul_ai · 2026-08-22
- Opinion: GPUs are massively underpriced given their intelligence value — Technical_Ad_6106 · 2026-08-22
- Scratch-built engine beats vendor runtime for ternary 8B models on free ARM cores — Annual_Manner_5901 · 2026-08-22
- Nvidia's New Vera CPU Tested: Big Bandwidth & FP8 Boost Agent Execution — pzakin · 2026-08-22
- Green Compute Launches Biogas-Powered GPU Cluster on Bittensor with 4090/5090 Rentals — markjeffrey · 2026-08-22
- SilkStack v2.2: local semantic search with custom WebLLM embedding model — skk80 · 2026-08-22