QCell: Recombining and Aligning Cell Queries for Overlapping Instance Segmentation
Yaroslav Prytula, Anton Popov, Dmytro Fishman
cs.CV, cs.AI, cs.LG
2026-08-29
QCell adds query recombination and DN-guided contrast on MaskDINO, hitting 65.9 AP and 78.6 AJI on ISBI2014 (+2.2/+2.7) plus a dense Organoids benchmark.
Overlapping cells in microscopy are not the same problem as occlusion in natural images. The hidden cell is still partly visible: contrast is weak, boundaries smear, and the intersection contains mixed evidence from both instances. Downstream morphology and density counts go wrong as soon as two cells are fused into one mask.
Most prior fixes stay inside a local region of interest. DoNet splits a cluster into intersection and complement regions and glues them back. BCNet predicts two layers with graph convolutions. AISFormer puts occluder, visible, amodal, and invisible mask tokens inside an RoI. None of them jointly reason over the whole scene about which evidence belongs to which neighbour. Shape priors that work on cars and people also fail here: cell morphology is too diverse for a learnable shape distribution.
QCell extends MaskDINO, a query-based segmenter in which a set of learned queries attends over the full image and each query is meant to own one instance. Under heavy overlap those queries pull toward each other through self-attention and collapse, so two cells become one mask.
The first add-on is instance recombination. Three light MLPs split every content query into amodal (full extent), visible, and invisible sub-queries. Each dots with pixel features to produce a mask, supervised by the matching ground-truth component. The three sub-queries are then fused into a refined query that must reconstruct the full amodal mask. A consistency loss asks that the refined mask equal the XOR of the visible and invisible predictions, with gradients flowing only through the refined branch. This is DoNet's decompose-and-recombine idea, moved from region proposals into query space, so there is no dependence on an upstream box.
The second add-on is contrastive query learning. Denoising queries, originally a DN-DETR trick that feeds noised ground-truth boxes into the decoder to speed training, are reused as anchors: DN queries of the same cell across denoising groups are positives, those of other cells are negatives. An InfoNCE term learns distinctive features; a cosine alignment term pushes different cells toward orthogonality. A DN oracle backs the design: swapping matched main-query predictions for the corresponding DN-query predictions at test time lifts ISBI AP from 63.7 to 68.6 and AJI from 75.9 to 80.0. DN queries really are cleaner representations, so using them as anchors is justified.
The total loss is MaskDINO's original objective plus recombination, discrimination, and alignment, summed over decoder layers. Backbone is ImageNet-pretrained ResNet-50; 100 queries on ISBI and Revvity, 300 on Organoids; one H200; three random seeds.
Three benchmarks: ISBI2014 cervical cytoplasm, where semi-transparent overlap is densest; Revvity-25 brightfield cancer cells; and a new Organoids set with 96 instances per test image on average and a max of 223.
| Method | ISBI AP | ISBI AJI | Revvity AP | Organoids AP |
| DoNet | 60.9 | 75.2 | 44.6 | 50.5 |
| MaskDINO | 63.7 | 75.9 | 52.3 | 49.7 |
| QCell | 65.9 | 78.6 | 52.9 | 51.0 |
Against MaskDINO, ISBI gains +2.2 AP, +2.3 F1, +2.7 AJI, and object-level false negatives drop from 11.6% to 8.7%. Revvity and Organoids move only +0.6 and +1.3 AP; AJI is essentially tied (73.6 vs 73.5, 63.2 vs 63.1). Parameter count is 45M and FLOPs 182G, up from MaskDINO's 44M/163G. Dice barely changes on any of the three sets, so the gain is instance separation, not boundary polish.
Ablations complicate the headline. Contrastive losses alone reach 67.0 AP on ISBI, higher than the full model's 65.9. The paper's account: recombination recovers heavily occluded cells the baseline missed, which helps AP50 and F1, while inferred hidden regions have only moderate IoU and drag AP75 and the threshold-averaged AP down. On the heavy-overlap subset (pairwise IoU ≥ 0.5 with another instance) the baseline is 11.67 AP and the full model 13.68. Absolute performance there is still poor.
For microscopy segmentation this is a clean transfer of amodal decomposition from RoIs into query space. Global attention is a better fit for semi-transparent overlap than two-stage detectors. Code is public. The Organoids density benchmark is worth tracking; the data is available on request.
For anyone training DETR-style models, DN queries as contrastive anchors travel: they already exist in the training loop and need no extra labels, and the oracle still leaves a 4.9 AP gap. The full model scoring below the contrastive-only variant is a useful warning. The two modules are not free to stack; the right mix depends on whether recall or precise IoU is the target.
There is no dedicated limitations section.
ISBI trains on 45 synthetic images, so the nicest numbers are the least trustworthy to generalize. Gains over MaskDINO on Revvity and Organoids sit around one AP; leaderboard rank is carried mostly by ISBI. Heavy-overlap AP of 13.68 means the core problem is not solved.
If a table only reports AP, recombination is a regression against contrastive-only. The backbone stays ResNet-50; no ViT scale-up. Organoids being "available upon request" adds friction for anyone who wants to follow. PCTrans is compared despite producing no per-instance scores, so its AP is not an informative baseline.