German bank's LLM compliance dilemma: DIY Langfuse vs Orq.ai for audit trails

CommonMinimum587 · reddit · 2026-11-19

A three-person ML team at a small German bank runs Llama 3.3 70B on vLLM across two A100s in their own server room for an internal policy assistant. When auditors asked which prompt and document version produced a wrong answer last month, they couldn't tell. They're choosing between a DIY stack (LiteLLM for keys/logging + self-hosted Langfuse for traces) and Orq.ai, which offers on-prem Helm deployment (even air-gapped), private vLLM endpoints, per-prompt version history, and audit logs on an enterprise plan with non-public pricing. They ask whether auditors accept self-hosted logs or demand a vendor to point at.

Original post →

More from Infra

Infra channel →