GLM 5.3 Flash kernels rewritten on RunInfra: 670 tok/s, 99.7% cache hit, AMD support
ycombinator · x · 2026-10-01
RunInfra spent September rewriting the inference kernels behind GLM 5.3 Flash and shipped the major release, hitting 670 tok/s on Vercel AI Gateway.
- Pricing: $0.11/1M input, $0.45/1M output, $0.03/1M cached tokens
- Context: 1M tokens, FP8 vendor-native build
- Hardware: now runs on AMD GPUs — same model, same API, more capacity
- Cache: 99.7% hit rate over the last 24 hours, with per-hit/miss logs in the dashboard
- Compatibility: OpenAI-compatible chat completions plus Anthropic-compatible /v1/messages, with image input, tool calling, JSON mode and streaming
- Privacy: zero data retention, never used for training
More from Infra
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16
- vLLM serves DeepSeek V4.1-Flash at 954K tokens/s per GPU, $38.96/GPU-hour on GB300 NVL72 — bookwormengr · 2026-11-16
- Colossus 2's 1.2M B200s could generate ~$400B/year in tokens at DeepSeek Flash rates — bookwormengr · 2026-11-16
- BIS issues rule blocking remote access to US AI chips via third-country data centers — teortaxesTex · 2026-11-16
- Qdrant 1.20.1: 50% fewer cold-storage reads, ~2x faster quantization, agent-ready setup — qdrant_engine · 2026-11-16
- One 16GB GPU runs songs, voice, images, 720p video and local RAG — all offline and open source — llama-of-death · 2026-11-16