FINSKILLOPS: A Self-Evolving Multi-Agent System for SEC Filing QA
Yanzhang Ma, Zhenghan Tai, Hanwei Wu, Sizhe Guan, Jianliang Lei, Hailin He, Chaolong Jiang, Jijun Chi, Tung Sum Thomas Kwok, Bohuai Xiao, Jingrui Tian, Xinlu Wu, Xingao Zhan, Peng Lu, Muzhi Li, Yihong Wu, Liheng Ma, Sicheng Lyu, Tianshuo Yan, Junhao Zhu, Yaqian Xu, Lei Ding, Yufei Cui, Ziquan Liu, Boyu Han, Hengli Liu, Ling Zhou, Xinyu Wang
cs.AI, cs.IR, cs.MA, cs.SE
2026-09-17
FinSkillOps turns SEC-filing failures into scoped skills with regression gates. Enhanced W-Corr. 3.70-4.55; 6 of 33 skills ship, monitoring non-correct falls 20.0% to 12.5%.
Financial QA systems usually freeze their reliability behavior at launch. Retrieval, prompts, and agent roles get tuned offline; after that, the same period mix-ups, entity confusions, and arithmetic slips keep showing up on new SEC-filing questions.
Self-improving agents can turn a failed trajectory into a new behavior. They rarely say where that behavior should fire, or which already-correct answers it might break. This paper treats post-deployment reliability as controlled skill maintenance: recurring failures become scoped natural-language patches, and each patch has to earn a frozen production snapshot.
FinSkillOps splits serving from evolution. Filings are parsed into modality-tagged units (narrative, rendered tables, section summaries). An orchestrator activates a subset of five analyst roles (general, quantitative, market, legal, company), retrieves along sparse, dense, and summary paths, and synthesizes a grounded answer. Skills enter the prompt at two points only: query decomposition, which constrains retrieval, and final synthesis, which constrains claims.
The offline loop diagnoses failures against the filing text in two LLM stages. Stage one reads the question, the failed answer, judge feedback, and retrieved context, then writes a grep plan. Deterministic grep runs that plan. Stage two reads the hits and emits a typed diagnosis, supporting snippets, and a propose / park / do-not-fix recommendation. Only modes that clear a frequency threshold become skill drafts. Each skill carries applicability conditions, behavior text, excluded cases, an injection point, and false-trigger guards.
Promotion needs three conditions at once: targeted-set Correct rate must rise, every previously Correct protected-suite case must stay Correct, and declared negative controls must not fire. Accepted skills enter a versioned registry; superseded versions go inactive and can later be retired. Serving uses a frozen snapshot. DeepSeek-V3.2 runs serving, diagnosis, and proposal; Qwen3-Max judges benchmark answers.
One frozen registry posts the highest verdict-weighted correctness (W-Corr., 0–5 with Correct=5, Partial=3, Incorrect=1, Failure=0) and reference consistency on all six benchmarks. On the 114-question enhanced in-house set (84 Zeekr, 20 Lotus, 10 NVIDIA), W-Corr. rises from a multi-agent init of 3.70 to 4.55. Zeekr moves from 3.46 to 4.52. Public-set gains are thinner: FinanceBench 3.49 versus FinGPT at 3.35, FinDER 2.88 versus init 2.68.
| Setup | Enhanced in-house | FinanceBench | FinDER | Protected regressions |
| Full system | 4.55 | 3.49 | 2.88 | 0 |
| No skill evolution | 3.70 | 3.15 | 2.68 | 0 |
| No protected gate | 4.32 | 3.38 | 2.79 | 3 |
| No failure taxonomy | 4.08 | 3.28 | 2.70 | 0 |
Ungated promotion records a larger targeted ΔAcc (+0.78 versus +0.62) and three protected regressions. Removing the period-alignment skill drops the in-house score to 4.21, the largest single-skill hole. Adapted AutoSkill and SkillOpt average 2.00 and 2.66 W-Corr. across four equally weighted sets, both below the 3.22 init; FinSkillOps averages 3.66 and wins 4/4.
A twelve-round operational study keeps evolution (300), protected (120), and monitoring (200) questions disjoint. Non-correct rates fall from 30.0% to 18.3% on evolution and from 20.0% to 12.5% on monitoring. Six of 33 proposed skills are promoted; five fail the protected set and expose seven regressions. From round 5 to 12 the registry holds four active skills, and static skill text shrinks from 421 to 347 tokens. Separate six-round runs on FinanceBench and SECQUE lift monitoring W-Corr. by 0.40 and 0.25.
For anyone shipping a financial agent, the transferable piece is the release process, not a bigger prompt. Twenty-seven of 33 candidates are refused, deferred, or withdrawn. Rejected drafts include a Table Row Lock that blocked valid cross-row aggregation and a Multi-Hop Decomposition skill that over-split simple questions and diverted retrieval. That is the point: natural-language patches misfire.
The portable kit is typed, evidence-grounded diagnosis; a protected suite plus negative controls; and a versioned frozen registry. Gains are large on the enhanced in-house set and modest on public benchmarks, which matches the failure modes the skills target (period, entity, calculation) rather than generic reading. Skill text is a small slice of prompt tokens. Serving cost sits near MoA, below FinDebate, and about 4× FinSage.
All evidence is SEC filing QA. The twelve-round run also changed routing and diagnosis, and the monitoring set was scored every round, so it tracks a deployment trajectory rather than the isolated effect of skill text, and it is not a blind generalization test. Candidate artifacts, admission logs, and split manifests are incomplete, which limits audit of gate decisions and item leakage. The financial-expert review path was unstaffed; admission rested on automatic checks. Judge agreement used annotators from the author group (four-way κ=.820/.675). On FinanceBench, FinGPT still leads four quality dimensions; on FinDER, FinSkillOps ties for lowest clarity. Conservative gates slow adaptation.