Creed: LLM Auditing That Infers Developer Beliefs Hits 0.78 Recall on 50 Fresh CVEs

LLM-based Code Auditing by Inferring and Evaluating Developer Assumptions

Junming Liu, Marcel Böhme

cs.CR, cs.SE

2026-10-12

Creed prompts an LLM to infer developer beliefs, check if they are reasonable and when they break; it reaches 0.78 recall on 50 recent CVEs, 10 points above the best baseline.

What problem this solves

Classic belief-based bug finding works like this: a null check implies the developer thinks the pointer may be null, so a later unchecked dereference breaks that belief. The beliefs are hard-coded as syntactic patterns over structures like the program dependence graph (PDG). That caps what it can find.

Many recent vulnerabilities need domain semantics instead. Take an authentication function that returns the same error for an unknown user and a wrong password, to stop user enumeration. An extra password-hash call on one path makes the response time differ, which leaks the same fact through a timing side-channel. No general syntactic pattern captures that. In a random sample of 50 recent CVEs, 40 were semantic and 21 of those could not be expressed as a general pattern.

Evaluation has its own trap. Function-level binary classification is shaky because whether a function is vulnerable often depends on its callers.

Method

Creed packs a human auditor's workflow into a single prompt, one LLM call per function.

• Belief inference. The model identifies actions at two levels: low-level ones (dereferencing a pointer) and the function's overall purpose (cancelling a payment). For each action it states the assumptions the developer seems to hold.
• Belief evaluation. A sanity check asks whether the belief is reasonable by standard security practice. An unreasonable belief is a bug regardless of context, for example treating a matching siteId as enough authorization to cancel a payment. A violation check handles reasonable beliefs: it derives the conditions under which they break, such as unequal response times.
• Reporting. Flagged beliefs are merged into findings, each with a root cause, the exploitation conditions and links back to supporting beliefs.

Because the analysis is function-level, results are only potential vulnerabilities. Callers may already mitigate them, so the conditions are spelled out for a human to check.

The paper also proposes two metrics. Recall is the share of known vulnerabilities for which at least one finding is consistent with the CVE description, fix commit message and patch. Overhead is the number of findings read per confirmed expected vulnerability. GPT-5.4 acts as judge, and every match it accepts was manually rechecked.

Results

The benchmark has 50 vulnerable functions from CVEs disclosed between September 2025 and March 2026, across 45 projects, 10 languages and 10 vulnerability types. Every approach uses Claude Sonnet 4.6.

• On paired vulnerable/patched functions, Creed matches on the vulnerable version but not the patched one in 33 of 50 pairs, the highest count.
• Looking only at the top 2 findings per function, Creed reaches 0.68 recall. At top 3 it reaches 0.72, above what any baseline gets with unlimited inspection.
• By type, Creed detects 90% of 10 memory-related and 75% of 40 semantic cases. Of the 21 non-expressible cases it finds 15, versus 12 for LLM-Thinking. Only Creed found the timing side-channel.
• Ablation: dropping action identification cuts recall from 0.78 to 0.60. Dropping either the sanity or the violation check cuts it to 0.66.
• Manual review of each function's first finding gives precision@1 of 0.74 (37/50), including 14 confirmed unexpected vulnerabilities.
• The judge agreed with two human annotators on 96% of 50 sampled findings.

Why it matters

For anyone building LLM code auditors, the transferable idea is the structure: make the model write down the assumptions behind the code, then challenge each one. With the same backbone this finds 5 more vulnerabilities than thinking mode (39 vs 34) at about 8.5k tokens and one call per function.

The evaluation recipe of explanations, recall and overhead is also reusable, since it avoids the caller-context problem of binary labels.

The gain is incremental, though: 10 points of recall over the best baseline, with higher overhead than VulTrial.

Limitations

• Fifty functions is a small benchmark, and the sample was filtered by hand to ensure the function holds the root cause. The paper gives no run-to-run variance.
• Recall relies on an LLM judge deciding that an explanation is consistent with the evidence. Human rechecks help, but the boundary is partly subjective.
• RepoAudit is a weak comparison: only 28 functions are in its supported languages and just 2 fall in its checker scope.
• Missing cross-function context causes false positives. Ten of 13 are issues already mitigated by a caller or guard API.
• The authors name two boundaries: application-specific policies (a user without download permission should not share) and open-ended violation conditions (unbounded dangerous substitutions). Of 11 misses, 3 never extracted the relevant action and 3 needed application-specific knowledge.
• A robustness run with DeepSeek V4 Pro is mentioned as also favoring Creed, but no numbers are given.

Terms

• Belief: an implicit assumption the developer relies on, such as that a generic error hides which case failed.
• PDG: program dependence graph, which records data and control dependencies between statements.
• User enumeration: an attack that uses response differences to learn whether a username exists.
• Timing side-channel: information leaked through differences in response time.
• P-C: in paired evaluation, a match on the vulnerable function and none on the patched one.
• LLM-as-a-Judge: using a separate LLM to decide whether an output meets a criterion.

Source

• arXiv 2610.15729v1
• PDF
• Hugging Face

What people are saying

• Creed: LLM code auditing infers developer beliefs to find semantic security flaws mboehme_

Related papers

• Same Qwen3 weights: 49% vulnerable code on the target GPU, 15.7% on the auditor
• Weak models harden agent benchmarks: hacker-fixer loop cuts Opus 5 attack success from 70% to 12%
• GPT-5.6-Sol fully solves only 31.5% of real-scale RE binaries
• Coding Agents Pass 91% Functionally but Only 25% Securely: 450 Real Vulnerabilities in LSCBench
• Opus 5 holds APEX-Harm attack success to 8%; script swaps still hit 84%
• One in Three 'Successful' LLM Vulnerability Patches Fails the Developer's Own Test

All paper explainers