The log fields this engineer records for every production LLM call, after getting burned
aureliatech · reddit · 2026-11-06
After being burned, the author shares the schema for logging every production LLM call, beyond the usual prompt+response:
Per call: request id and parent id (to stitch multi-step flows), feature name and code version, model name and exact version, prompt template id and version (not just rendered text), input/output tokens and latency, retry count and errors, raw and parsed outputs stored separately, parsing/validation failure flags, hashed user/tenant id.
Per outcome, backfilled: whether a human accepted, edited or rejected it, and the final version if edited — the most-skipped field that later becomes your eval set.
Two notes: watch retention and PII before storing raw prompts, and keep template id separate from rendered text or you can't trace which prompt change caused a regression.
More from coding & agent
- Drew Breunig's QCon Talk Breaks Down Prompt Debt and Four Context Failures With Concrete Fixes — dbreunig · 2026-11-06
- Dev essay: why a runaway intelligence explosion isn't the risk — 'early onset decoherence' is — round · 2026-11-06
- Sardine launches Know Your Human protocol to verify humans behind shopping agents — astrange · 2026-11-06
- Indie dev cuts 5,672 lines rebuilding trip planner's Days screen into one-drag editing — SeanOliver · 2026-11-06
- LangChain details optimizing browser use for deep agents; design agent fixes its own blog formatting — LangChain · 2026-11-06
- Baker Lab + NVIDIA unveil RFD4-Proteina: agents run protein design tool calls on GFP — AllThingsApx · 2026-11-06