Testing a Langfuse release gate: one missing exported score correctly blocks the passing receipt
MongooseWonderful198 · reddit · 2026-11-17
The author of MIT-licensed Langfuse Prompt Release Tools (Python/JS) added a test in the 0.8 alpha for a specific release check: when all local evaluations pass but one required run-level score is absent from Langfuse, the release importer must refuse a passing receipt. The demo runs three synthetic cases on disposable servers — a complete export gets confirmed, while withholding one score leaves the result pending after flush() and bounded polling, and the verifier refuses. The verifier checks captured experiments, datasets, required cases, and score types/values. This tests the toolkit's refusal path, not a Langfuse bug — useful if your release policy requires stored evaluation evidence before trusting a passing local result.
More from coding & agent
- Five-stage agent routine that stopped an LLM deleting my repo — with the Brief prompt to steal — Iiifebri · 2026-11-17
- PlugIt turns any OpenAPI spec into a curated, read-only-by-default MCP server in a Go binary — Low_Strawberry2484 · 2026-11-17
- Amp vs Claude Code hands-on: user finds Amp gets more done even as $20-tier Fable 5.2 access ends — bytebot · 2026-11-17
- supermemory says 4 unicorns adopted its AI memory layer without ever talking to the team — julianweisser · 2026-11-17
- He vibecoded a whole bank with one prompt: "Make a bank. no bugs. make it safe." — martyamark · 2026-11-17
- Git Hook Speed Test: Rust-Based hk Wins at 1.37s, Python pre-commit Slowest — aniketmaurya · 2026-11-17