MCP server returns parquet handles instead of rows, cutting 3.2M tokens to 1,700
Daniel_Aperiodic · reddit · 2026-11-17
Aperiodic's founder details how their MCP server handles big market datasets: one month of 5-minute flow data for a single symbol is 8k+ rows by 35+ columns — roughly 3.2M tokens as JSON, far beyond any model context.
Their fix: tools return a lightweight 'handle' instead of data. getdata returns signed parquet URLs (valid 1 hour), schema, row counts, per-column min/max/null stats read from parquet footers, five sample rows, and a polars loading snippet — about 1,700 tokens total. The agent's code downloads the file; the model never sees the rows.
For clients that can't run code, a summarize tool computes stats, correlation matrices, or resamples series to ≤500 points server-side; search/fetch exist to match ChatGPT connectors' expected shape. The author admits Glama scored the tool definitions 3.6/5 due to mixed naming conventions, and openly asks for better solutions for sandboxes that can't fetch external URLs at all.
More from coding & agent
- 'Stuck at 99%': a single-file HTML toy progress bar with fully synthesized Web Audio — prasenx · 2026-11-17
- The constellation view: monitoring agents as stars — green, yellow, red, gray — kevinkern · 2026-11-17
- Directing AI agents by playtesting: one note turned an ugly lighthouse into a beauty — chrisfirst · 2026-11-17
- 943 agents, 11.5B tokens, $5,900: 97% of tokens went to re-reading context — chrisfirst · 2026-11-17
- Dev builds stunt-driving game in 4 days with 943 AI agents, 11.5B tokens and $5,900 of compute — chrisfirst · 2026-11-17
- Monid launches 'Costco for Agents': 3,000+ APIs, pay-per-call, zero markup — socialwithaayan · 2026-11-17