Building an AI data analyst: how to handle conflicting per-user KPI definitions
Lopsided_Feedback_81 · reddit · 2026-11-11
A developer of an AI data analyst product for Shopify stores describes a real engineering pain: users within the same company define KPIs differently (e.g., is revenue net of refunds?), rarely state their definition, and blame the model when results mismatch.
Current approach: run a cheap model daily over conversation traces in MLflow, extract explicitly stated definitions, store them in a Postgres jsonb keyed by userid, and inject them into the system prompt at session start. Three problems: it's slow, misses implicit definitions, and may mistake one-off statements for durable definitions. The post asks the community for better designs.
More from coding & agent
- Open-source Agent Standup taps Claude's new Whiteboards and Sheets for agent standups and timesheets — stickygoose · 2026-11-11
- Nod details its agent evals: 98% task completion across 678 runs, zero of 216 memory injections succeed — rohanpaul_ai · 2026-11-11
- 9-year Rust veteran argues Zig beats Rust in the AI coding era — nicolascraske · 2026-11-11
- Cheap builder + frontier reviewer loop matches flagship on 66 tasks at ~1/16 the cost ($0.05 vs $0.72) — KrystalUnity · 2026-11-11
- As Hand-Coding Fades, the Edge Shifts to Research Insights LLMs Can't Supply — janusch_patas · 2026-11-11
- An LLM Hallucination Delivered 72 Eggs; Lesson: Always Review the Plan First — JFPuget · 2026-11-11