Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs
Xiangfan Wu, Zonghao Ying, Huiyu Wu, Xing Zheng, Huangsheng Cheng, Xiaorong Shi, Jing Guo
cs.CR, cs.AI
2026-08-17
Modeling vendor routing as a stochastic process, Ventor-QTest audits hosted LLM APIs from returned text counts alone, reporting average fidelity loss (AFL) and extreme fidelity loss (EFL). All six third-party DeepSeek routes showed deviation, and the highest-EFL route's Terminal-Bench pass rate fell from 82.6% to 13.6% across exposure quartiles.
When you buy a third-party hosted API for an open-weight model, you trust the model name on the vendor's page. The vendor's degrees of freedom are wide: a different inference engine, batching strategy, or numerical kernel; undisclosed FP8 or FP4 quantization; a replica pool with altered configs mixed in; or outright substitution with forged metadata. The nastiest case is adaptive selective routing: recognize audit traffic and serve the faithful model while normal traffic gets the cheaper stand-in. Nobody had systematically measured this ecosystem; the paper builds a black-box instrument that requires no vendor cooperation.
The hard part: a black-box audit gets no logprobs, only returned text. Sampling is stochastic by design, so how do you tell normal jitter from routing to a different model?
The authors model a route (a snapshot of one serving endpoint within a time window) as a stochastic process and split the audit into two components, each reporting one statistic:
Why both? AFL sees typical behavior, EFL sees tail behavior, and they need not order routes the same way. One route in the experiments shows low AFL with pronounced EFL, which the mean alone would miss.
Validation against a comparator: on three logprob-capable routes, text-count AFL agrees with a logprob-derived coarsened KL at Pearson r=0.971, evidence that the black-box statistic measures what the white-box one does (descriptively; with three routes the exact permutation test is not confirmatory).
Auditing seven route snapshots of DeepSeek V4 Flash 0731 (an official self-check plus six third parties): the official route is clean, Sr=-0.0007 with 95% CrI [-0.0144, 0.0193], p=0.515. All six third-party routes deviate significantly:
| Route | AFL (Sr) | 95% CrI |
| Aliyun 0731 | 0.5704 | [0.4710, 0.6765] |
| StreamLake | 0.2950 | [0.2364, 0.3533] |
| Baidu Qianfan 0731 | 0.1875 | [0.1456, 0.2365] |
| Ark 0731 | 0.1591 | [0.1129, 0.2071] |
| DigitalOcean | 0.1250 | [0.0789, 0.1766] |
| DeepInfra FP8 | 0.1185 | [0.0782, 0.1611] |
All six have Holm-adjusted p=0.00035. On the EFL side DigitalOcean carries the heaviest tail: median absolute drift 0.00876, SD 0.01676, and a maximum of 0.05960 across twenty 500-position runs, versus 0.00545/0.00835/0.03498 for the official route. StreamLake is the converse, smallest median but second-most-pronounced SD, confirming AFL and EFL measure different things.
The downstream comparison is the most interesting finding. GPQA-Diamond accuracy spans 69.7-74.7% across the seven routes with overlapping Wilson intervals and no detectable association with AFL or EFL. But on Terminal-Bench (89 long-horizon agentic tasks per route), DigitalOcean, the route with the most pronounced EFL, drops from 82.6% pass rate in the lowest task-exposure quartile to 13.6% in the highest, a 20.6-point gap (bootstrap 95% CI [-36.0, -4.5], one-sided p=0.0059). The reading: every step of a long task is another model decision, so longer trajectories offer more opportunities for one extreme deviation to sink the run. Multiple-choice-style short tasks are insensitive to that.
Teams buying third-party APIs can use this directly: the tool is open-sourced under Tencent's AI-Infra-Guard repo at services/apichecker/ventorqtest. The two-statistic finding has an operational punchline: clean short-task metrics do not mean your agent workload is getting the model you paid for. Teams running agents should watch EFL, not just averages. For procurement and oversight, this is the first open-source audit that formalizes adaptive selective routing in its threat model with a reproducible measurement.
The paper states its own boundaries carefully: AFL holds only within the declared category map and observed window; EFL is a centered-surprisal deviation, not a KL estimator, and provides no universal ordering of fidelity; empirical quantiles from twenty runs are descriptive; multinomial calibration assumes independent requests, and correlated routing plus reference uncertainty can make intervals optimistic. The downstream association is exploratory, with probe and benchmark windows unsynchronized, so no causal attribution. A provider that perfectly recognizes audit traffic remains undetectable.
My reservations: all seven routes are snapshots of one model (DeepSeek V4 Flash) in one time window, so whether AFL values (Aliyun's 0.57, say) compare across models and vendors is unanswered. The reference distribution comes from the official API; if the official endpoint itself changes undisclosed behavior, the audit's anchor drifts.