Classification by Attention: Scene Graph Classification with Prior Knowledge
Sahand Sharifzadeh, Sina Moayed Baharlou, Volker Tresp
AAAI Conference on Artificia
cs.CV, cs.LG
2020-11-20
Schemata turns scene-graph classification into attention over class embeddings and injects them as prior. With 1% labels plus BYOL, PredCls R@100 hits 65.68 vs 65.7 on full data.
Scene graph classification labels objects and the relations between them as (head, predicate, tail) triples. Lighting, pose, and occlusion make the same class look different across images. A bowl that is almost fully hidden has almost no visual evidence left.
Two families of fixes were common. One passes messages over every object's visual features in the image; the space of neighbor appearances explodes, so co-occurrence is hard to learn stably. The other trains a separate prior, either co-occurrence counts or a knowledge-graph embedding, then multiplies it with the visual scores. A horse and a donkey look similar in pixels, but a triple store may only contain (Woman, rides, Horse). The prior never sees that visual neighborhood. Perception and common sense do not share parameters, and they cannot use the current image to choose between "fruit in a bowl" and "fruit on a tree".
The LMU Munich, Sapienza, and Siemens group folds perception and prior into one network. Each object class and each predicate class gets a learned vector, a schema, that is supposed to hold both appearance and how that class participates in relations.
Classification is an attention layer, not a fully connected head. After a Graph Transformer contextualizes node features, each node dots with every schema; the softmax weights are the class scores. The weighted sum of schemata is added back to the original image features, not to the previous Graph Transformer output, so the raw visual signal does not wash out. The same Graph Transformer then messages again and reclassifies. One such loop is an assimilation. The model trains with 4 loops and is tested with up to 8. Scheduled sampling replaces up to 10% of false negatives with the true label so the net does not overfit one round's error rate.
The Graph Transformer is a Graph Network Block variant: 4 layers, 5 heads each. Object nodes neighbor their predicate nodes; predicate nodes neighbor head and tail. Training is multi-task. The first loop is image-only classification (IC). Later loops condition on the previous predictions (ICP). With no image, visual features are zeroed and the attention coefficients are replaced by a one-hot, so a list of triples can fine-tune ICP alone. In the low-label setting the backbone is ImageNet-pretrained, BYOL-tuned on all Visual Genome images, then supervised on the 1% or 10% boxed split.
The split is the usual Visual Genome one: 150 object classes, 50 predicates, about 11.5 objects and 6.2 predicates per image. Full-data runs use Faster R-CNN with VGG-16 to match prior papers.
Under the unconstrained protocol Schemata's mean Recall is 68.7, against CCMT at 68.4 and KERN at 66.4. SGCls R@50/100 is 48.5/52.1; PredCls is 83.8/90.5. The long tail moves more: unconstrained mean Recall is 36.3 versus KERN's 32.8. Constrained mean is 53.5 versus CCMT's 53.3, essentially tied.
The low-label table is the result that matters. Backbone is ResNet-50; metric is R@100; numbers are means over five random splits:
| Setup | Labels | SGCls | PredCls | Object cls. |
| Self-sup + IC | 1% | 12.12 | 48.10 | 40.75 |
| Self-sup + IC + ICP | 1% | 15.36 | 65.68 | 42.09 |
| Self-sup + IC + ICP | 100% | 37.1 | 65.7 | 68.4 |
Against self-sup + IC at 1% labels, SGCls rises about 27% relative, PredCls about 36%, object classification about 3%. PredCls 65.68 already sits on the full-data 65.7. Schema-only link prediction with zero image features (Schemata-PKG) scores unconstrained PredCls R@50/100 of 48.9/54.2, so the class vectors do hold a commonsense graph.
A 0.3 mean-Recall edge over CCMT is not a reason to reopen 2020 scene-graph leaderboards. The inductive bias is: the classification weights are already embeddings that can do link prediction, and the prior is mixed in embedding space rather than multiplied in probability space. Predicates live on co-occurrence; a triple list can nearly replace image labels for that head. The backbone is dated. The structural idea, classify by attending to prototypes then inject those prototypes as messages, is the same family as prototype networks and later prompts.
There is no Limitations section. Full-data SGCls is tied with CCMT (unconstrained R@50 48.5 vs 48.6). SGDet is claimed to move like SGCls; the numbers live in a code repo, not in the paper. The "external triples" in the 1% study are the Visual Genome labels that were dropped from that split, not a foreign KB such as ConceptNet, so the wording oversells transfer. Object classification and SGCls at 1% remain far from full data, 15.36 vs 37.1. Boxes are given. A Graph Transformer could in principle learn a prior on its own; the extra assimilations are not ablated against "more graph-conv rounds, no schema injection", so how much of the gain is the explicit bias versus extra compute is not pinned down.