EpiCon: Collective Agent Learning through Co-Evolving Multimodal Memory
Ziyun Zeng, Hang Hua, Shaden Alshammari, Rogerio Feris, William T. Freeman, Jiebo Luo
cs.CV
2026-09-30
EpiCon's two 2B models share multimodal experience without host updates. Four setups gain 1.7-4.9 macro points over No Memory; memory time drops 67-74% vs backbone-sized models.
A multi-agent run leaves procedures, failures, and the evidence it used. External memory can feed that into later inference without touching host weights. Reuse across systems is the hard part. Different harnesses and backbones rarely read the same store. On multimodal items the useful signal is often a region of an image, which text notes do not carry. When a question is retried, the written guidance and the crop have to move together, or the next attempt stares at the wrong patch.
Prior work split these pieces: linked notes, hierarchical experience, shared memory across models, multimodal evidence. EpiCon wires two loops. Inside a question, text and crops co-evolve from feedback. Across questions, lessons sit in a shared tree that another agent stack can query and write into.
Memory lives in a harness outside the MAS. Host orchestration and parameters stay put. Two independently fine-tuned Qwen3.5-2B models do the work: a Memory Controller for one question, a Tree Self-Organizer for the shared bank.
Question-level memory is a pair: text guidance plus a region on a source image. The controller sees the question, original images, prior memory, latest answer, and the execution trace, then jointly proposes new text, whether to use vision, and which box to crop. The harness validates and crops; invalid proposals leave memory unchanged. A gate decides whether the next solver input includes the visual crop. Turning the gate off does not delete stored vision.
Think of a sticky note and a screenshot that both get rewritten after each try. If the note now says to transfer at Termini, the crop should move to the interchange, not stay on the half of the map that already failed.
Across questions the bank is a tree. Leaves hold item-specific lessons; internal nodes group and summarize. During maintenance the organizer can Place, Merge, Split+Lift, or Consolidate. Retrieval first recalls candidates with a frozen text encoder over topic and question anchors, then the organizer picks a rule, a concrete lesson, or nothing. At evaluation the historical bank is frozen and question-level updates never write back.
Supervision comes from MAS traces on visual math, documents, and charts. Qwen3.8-27B is the teacher; Qwen3.8-Flash-Next revises failed targets. The controller keeps about 25K replay-checked transforms: Repair turns a failed attempt into a success; Compress keeps success while shortening text and not growing crop area. The organizer has about 6.5K tree operations that pass schema, node-reference, and provenance checks; those operations were not replayed through the MAS. Both 2B models train with LoRA (rank 16) for three epochs, visual encoder and aligner frozen.
Evaluation uses 4,538 questions from 11 benchmarks in four domains: document understanding, vision-to-code, vision-grounded math, and general visual-language reasoning. Codex and DeepSeek-Harness each pair with Qwen3.8-27B and Gemma4-31B. Every memory method builds its bank from the same 1,000 construction questions, disjoint from eval. The main table is one solve per question with question-level updates off, so the score is historical reuse, not extra retries.
Backbone-sized EpiCon leads the eleven-benchmark macro-average in all four hosts, 1.9 to 5.9 points above the strongest of Mem0, Cognee, A-Mem, and Agent-KB (those systems run memory ops zero-shot on the same backbone). On Codex + Qwen3.8-27B, ParseBench moves from 57.1 (No Memory) to 68.5 and MATH-Vision from 20.4 to 48.6.
The trained 2B pair cuts memory-operation time 67% to 74% and total (MAS + memory) time 25% to 35%. Macro scores sit 0.9 to 3.6 points below the backbone-sized variant, and still 1.7 to 4.9 above No Memory. With Gemma4-31B, the backbone variant is +7.0 on Codex and +2.7 on DeepSeek-Harness; the 2B variant is +4.2 and +1.7. WorldBench rises in all four hosts (+3.2 to +7.8). ReasonMap (+0.5 to +1.9) and BabyVision (-2.7 to +4.8) do not.
| Comparison | Metric | Number |
| 2B vs No Memory, four hosts | 11-benchmark macro | +1.7 to +4.9 |
| 2B vs backbone-sized memory models | memory-op time | -67% to -74% |
| Frozen Codex+Qwen bank, new backbone / harness | macro | +3.2 / +4.2 |
| Other harness evolves bank, original system re-solves | macro | 53.5→55.7 / 53.4→56.6 |
| Tree vs flat bank | ParseBench | 67.3 vs 52.2 |
Transfer is the result that matters. A bank built by Codex + Qwen, frozen, lifts Codex + Gemma by 3.2 macro points and DeepSeek-Harness + Qwen by 4.2, still one solve. Let the other harness evolve that bank on a second 1,000 questions and the original system, on 4,388 held-out items, goes 53.5 → 55.7 (Codex) and 53.4 → 56.6 (DeepSeek-Harness). Gains are uneven: Codex ParseBench falls 63.4 → 58.9 after DeepSeek writes into the bank; DeepSeek MATH-Vision falls 56.6 → 53.4 after Codex writes.
With the bank off and up to five attempts, the 2B controller raises all four ablation benchmarks; Codex MATH-Vision goes 20.4 → 41.2. Evolving vision plus adaptive injection beats frozen crops and always-on injection, and cuts Codex MAS time by about 25%. Tree vs flat, same lessons: ParseBench 52.2 → 67.3. On Codex + GPT-5.6-Luna, with the 2B modules and bank frozen, all four reported scores rise (ParseBench 63.8 → 66.5, MATH-Vision 51.2 → 57.2).
Memory is a plug-in service. Hosts only consume retrieval. A 2B pair that is 67-74% faster on memory ops is closer to something you would attach to several agent stacks than running 27B or 31B for memory too. Frozen-bank transfer across harness and backbone means a new stack can borrow an existing store instead of filling one from scratch.
The useful setting is narrow: documents, charts, visual math, anywhere the answer lives in a patch of the image and the same task family repeats. WorldBench is stable; ReasonMap and BabyVision are not. One-shot Q&A with no accumulating task type pays for a tree and co-evolution it will not use. Against Mem0, A-Mem, and Agent-KB this is a solid incremental join of multimodal co-evolution and cross-system sharing, not a new learning regime.
There is no Limitations section. The appendix states two hard constraints: tree operations are checked for structure, not replayed through the MAS; controller replay certifies a transform on the source task, not on later tasks.
The main table correctly strips retries from historical retrieval. The controller ablation does not: No Memory gets one attempt, MC gets up to five, so the MATH-Vision jump from 20.4 to 41.2 mixes memory with extra tries.
Construction and eval come from the same benchmarks, only the items differ. That is near-distribution memory. Training screens with reference answers; online updates do not. Cross-harness expansion can hurt the original system on individual boards, with no conflict arbitration. On Codex + Qwen, 2B MP-DocVQA drops 79.5 → 66.1; the macro gain is carried by MATH-Vision. DeepSeek + Gemma sits at 2.7 MP-DocVQA with No Memory, so the +1.7 macro on that host is a weak number.