2026-09-24
A symbolic agent enumerates valid query plans and the LM only scores them: BERT-base lifts GraphQ F1 from 34.3 to 52.0, and Codex does few-shot KBQA with 10 examples.
The dominant way to ground language models in an environment, whether a robot, an API surface, or a database, is to have them generate the executable plan directly. That piles three responsibilities on the model: grammaticality, faithfulness to the environment, and controllability. It breaks as environments grow: Freebase holds 45 million entities and 3 billion facts, most model-generated queries simply do not execute, and in low-data regimes the outputs of generative systems like UnifiedSKG are mostly invalid programs. This ACL 2023 paper (Yu Gu and Yu Su's group at Ohio State) flips the division of labor: a symbolic agent generates valid plans, and the language model only scores them.
Pangu loops four steps: the agent extends current plans into a batch of new candidates, each guaranteed executable; the LM scores how well each candidate matches the user's utterance; the top-K survive to the next round; a termination check ends the search. Instantiated on knowledge base question answering, plans are S-expressions; the agent executes partial queries and enumerates the relations attached to intermediate entities to grow candidates. The scoring interface adapts to the model: BERT reads its [CLS] representation, T5 decodes one step and reads the probability of an unused token, Codex yields P(plan|utterance) with length normalization.
The scorer only has to learn a partial order: gold subplans above wrong same-length plans, complete plans above incomplete ones. That is far easier than learning to generate whole plans from scratch.
| Dataset | Prior best F1 | Pangu |
| GrailQA | 78.7 (DecAF) | BERT-base 79.9, T5-3B 81.7 |
| GraphQ | 34.3 (ArcaneQA) | BERT-base 52.0 |
| WebQSP | 78.8 (DecAF, 3B) | BERT-base 77.9, T5-3B 79.6 |
GraphQ jumps 17.7 points. In the few-shot setting, Pangu with Codex and 10 examples reaches 42.8 F1 on GraphQ, beating ArcaneQA fine-tuned on the full training set (34.3); on GrailQA, 10-shot gives 56.3, and EM passes 50% with a single example. This was the first demonstration that LLMs can do KBQA in-context at all.
The paper also explains why generation struggles: ArcaneQA's output probability distributions shift drastically between programs seen and unseen in training, while Pangu's stay consistent. Autoregressive models overfit seen structures and crack on novel compositions; scoring generalizes more naturally.
This is the cleanest early validation of the recipe "LLM as scorer, symbolic system as executor" in grounded language understanding: a 110M-parameter BERT-base matches or beats generative systems ten times larger, and few-shot capability comes along for free. The same recipe resurfaced later in tool use, beam-search code generation, and process reward models. For agent builders the reminder stands: generation is not the only way to use an LM, and over enumerable action spaces scoring is often the stabler one.
The authors concede the compute cost: iterative scoring makes training and inference about twice as slow as ArcaneQA with double GPU memory, and forward passes scale with the number of candidates. Few-shot prompting remains crude, and the gap to fine-tuning is not closed. A boundary worth stating: the recipe needs an enumerable space of valid plans, and KBQA is unusually tidy; in open environments the enumeration itself becomes the bottleneck, and the paper validates only on KBQA.