First Oracle PL/SQL benchmark, ported from MBPP/MBPP+, to be presented at EMNLP
srchvrs · x · 2026-10-06
srchvrs and team built the first Oracle PL/SQL code benchmark, to be presented as a poster at EMNLP Industry Track.
Key details:
- MBPP and MBPP+ were "translated" into PL/SQL via rule-based mapping, with unit tests converted alongside quality checks to ensure the translation is nearly fault-free
- The author confirms the MBPP+ PL/SQL harness has no faults: every sample has a replay function satisfying all test cases
- A score of at least 87% on MBPP+ PL/SQL is known to be achievable
The benchmark fills a gap in code evaluation coverage for enterprise SQL dialects, enabling tests of how models and agents handle simple programming tasks in PL/SQL.
More from Research
- Arc Institute unveils PIE, a perturbation model that generalizes to unseen cell lines and perturbations — yusufroohani · 2026-10-06
- Two bottlenecks holding back perturbation models: system representation and signal identification — yusufroohani · 2026-10-06
- Apple researcher lands two NeurIPS 2026 papers on normalizing flows, including Normalizing Trajectory Models — thoma_gu · 2026-10-06
- NormalFactory unveils FrontierCAD, a verifier for grading AI-generated CAD parts and assemblies — hudzah · 2026-10-06
- SemanTok: 49M autoregressive video model beats SOTA rivals 47x its size — CSProfKGD · 2026-10-06
- Kundaje: current experiment designs can't yield virtual cells or causal world models — anshulkundaje · 2026-10-06