Exceeding the Limits of Visual-Linguistic Multi-Task Learning
Cameron R. Wolfe, Keld T. Lundgaard
cs.AI, cs.CL, cs.CV, cs.LG
2021-07-28
A shared multimodal BERT on 1,000 shop taxonomy tasks lifts mean accuracy from 61.5% to 75.0% and low-resource accuracy from 34.4% to 57.1% versus per-task models.
Visual-linguistic multi-task learning in 2021 usually tied together a dozen tasks. BERT was treated as overparameterized enough, yet nobody had pushed a shared backbone to hundreds or a thousand classification heads. The industrial case is blunter: one SaaS stack faces hundreds of customer sites, inputs look alike (title, description, product photos), label taxonomies do not, and small sites do not have enough labels to fine-tune alone. Cameron Wolfe, then a Rice intern at Salesforce Einstein, turns live multi-tenant catalog data into what the paper calls large-scale MTL: one shared model, at least 100 tasks, up to 1,000.
The data stay private. The 100-task set is product type and category from 50 sites, more than 250,000 products, correlated because apparel trees look alike. Larger sets add sites; reported metrics still cover only the original 100 tasks so the test distribution does not move.
The backbone is Kiela et al.'s single-stream multimodal BERT base, started from public pretrained weights. Text follows BERT tokenization. Images go through a frozen EfficientNet-B4, then a learned linear map; all image embeddings and text tokens form one sequence, with different token types for vision versus language and a shared position id for unordered photos. BERT weights are shared; each task has its own head, and the task id is part of the forward pass.
The 100-task run lasts 15 epochs, batch 64, one task per batch, AdamW, one V100. Convergence needs a warmup: learning rate climbs from \(10^{-5}\) to \(10^{-4}\) over four epochs, then drops 10× at epochs 8 and 12. A tiny fixed rate also converges, slowly; freezing the backbone first is a bit better; warmup wins.
Task sampling uses \(P(T)\propto NT^{\alpha}\). \(\alpha=1\) follows dataset size, \(\alpha=0\) is uniform. Exponential decay from 1.0 to 0.1 eats large tasks early and lifts small ones later. Ten consecutive steps on the same task diverged on the 100-task set, so every step draws a new task.
Fully connected heads cost 58 million task-specific parameters at 100 tasks. Projecting BERT outputs to a small \(dt\) and running self-attention there, with \(dt=64\), cuts that by about 10× at similar accuracy. Large heads help high-resource tasks and overfit low-resource ones. DyPA uses label count as a complexity proxy and assigns \(dt=128/256/512/1024\) by quartile, so big tasks get bigger heads without blowing up small ones, still about 3.5× fewer task parameters than fully connected heads.
Every number below is on the original 100-task 80/20 hold-out.
| Method | Mean acc. | T10 (high-resource) | B10 (low-resource) |
| Per-task BERT | 61.50% | 72.33% | 34.40% |
| 100-task MTL | 73.29% | 69.30% | 54.39% |
| 500-task MTL | 72.54% | 67.36% | 52.80% |
| 1,000-task MTL | 74.98% | 67.02% | 57.08% |
Separate models win T10 by about 3 points; those tasks each have more than 10k labels. Bottom-decile separate training sits under 35%; the 1,000-task model raises B10 by 22.7 points. One thousand independent BERT bases would be about 110 billion parameters; the shared model is about 250 million.
Using the MTL backbone as initialization on a 405,840-example, 2,196-class cross-site ontology: BERT base 90.27%, 100-task MTL 90.77%, 1,000-task 91.12%. Swapping in BERT large as the shared trunk yields 73.39% mean on 100 tasks and loses on both T10 and B10 versus base. Adding an ontology task an order of magnitude larger than any original task raises T10 by about 4 points; sampling and DyPA do not collapse. On 10/25/50/75-task slices, \(\alpha\)-decay plus DyPA always wins, by roughly 4.5–9 points.
This is a recipe a multi-tenant classifier can actually run: shared BERT, a new task every step, exponential \(\alpha\) decay, capacity allocated by data size. Small shops borrow inductive bias from large ones; large shops give up about 3 T10 points. The 2021 setting is the easy slope for MTL, correlated tasks with isomorphic inputs. It is not evidence that 1,000 unrelated tasks will pack the same way.
The data are private, so nobody outside can reproduce or inspect label noise. Almost every task is an e-commerce taxonomy, highly correlated; transfer to a heterogeneous suite like GLUE is untested. The vision trunk is frozen and photos are unordered. Metrics are mean per-task accuracy, with no calibration, tail F1, or fairness. The 1,000-task model is still tested on the original 100 heads; the extra 900 tasks are training signal only. BERT large did not help, so the capacity story stops at "base is enough" without a finer overfitting diagnosis. The authors flag heterogeneous and adversarial tasks as future work.