Hand Visibility Detector: Per-Keypoint Visibility Estimation for Hands
Ryosei Hara, Masashi Hatano, Rintaro Yanagi, Atsushi Hashimoto, Takuma Yagi, Mariko Isogawa
cs.CV
2026-08-12
Per-joint hand visibility as a standalone task: a frozen WiLoR pose backbone plus a tiny head hits 0.931 mAP on HInt and cuts triangulation error up to 10%.
Hand pose estimation (HPE) underpins AR/VR and robotics, but most models emit a 2D or 3D coordinate for every joint without saying whether that joint was actually visible or merely guessed under occlusion. For hands, which are constantly self-occluded, blocked by grasped objects, or clipped at the image edge, that distinction matters: a downstream pipeline that trusts an invisible joint's coordinate propagates error across the whole hand.
Visibility had been used only as an auxiliary signal to help the main pose model. This paper promotes it to a standalone task: predict, for each joint, a [0,1] confidence of being directly observable. The authors call it the first systematic study of per-joint hand visibility estimation.
Two pieces: a frozen ViT backbone taken from a pretrained HPE model (HaMeR or WiLoR), and a visibility head of only 0.83M parameters, 0.131% of the 631M-parameter whole.
Freezing the backbone is the central choice. Judging an occluded joint from its own location is hopeless; you need the whole-hand structure to reason about what is hidden. Large-scale pretrained HPE models have learned to recover 3D hand shape from millions of occluded images, exactly the structural prior needed. Freezing preserves it. Fine-tuning WiLoR in ablation drops mAP from 0.931 to 0.622; task-specific tuning corrupts the features.
The backbone choice matters too. Hand-specific models (HaMeR, WiLoR) beat general image encoders; the 631M WiLoR surpasses even the 840M DINOv3. The visibility head compresses features with 1x1 convolution, runs a Gated Attention Unit (GAU) for global spatial dependency, then projects to 21 MANO joint channels and spatially pools to per-joint logits. Removing the GAU drops mAP from 0.931 to 0.887. The loss is plain per-joint binary cross-entropy. Training takes about 2.5 hours on a single H200 with roughly 10 GB of memory.
On HInt (25,273 training frames, 5,374 evaluation frames, with manual per-joint visibility labels):
| Method | mAP | F1 |
| Kim et al. (2021) | 0.895 | 0.858 |
| Contact4D (2026) | 0.897 | 0.860 |
| Ours | 0.931 | 0.896 |
That is 3.4 mAP points over the strongest baseline. F1 stays above 0.88 across a binarization threshold of 0.3 to 0.7, peaking at 0.5, so the method is insensitive to the threshold.
The downstream test is multi-view triangulation. Weighting triangulation by per-joint visibility beats both unweighted and detection-confidence weighting on DexYCB, HO3D, and H2O, lowering the median, mean, and interquartile range of reprojection error (pixel error from reprojecting reconstructed 3D joints back into each view). The largest gain is on HO3D, which has fewer views and heavier object occlusion: mean reprojection error falls 10.1%.
For teams building hand-based applications this is a plug-in reliability switch. Every joint coordinate from a pose estimator can now carry a "trust this much" score, and downstream code can downweight or drop low-visibility joints. Multi-view pipelines that batch-annotate 3D hand pose are the direct beneficiaries; visibility-weighted triangulation measurably reduces reconstruction error. Code is open source.
For purely monocular settings without multiple views, the downstream gain is not validated directly. The larger contribution is establishing visibility as an independently evaluated task rather than an internal auxiliary quantity.
The authors acknowledge only one: the method processes frames independently, with no temporal consistency, and video input is left to future work.
A few gaps remain. Evaluation is concentrated on the single HInt dataset; cross-domain generalization across cameras, populations, and occlusion types is not reported. The 10.1% downstream gain is pixel-level reprojection error, not accuracy on an end task such as grasping or gesture recognition, so "10% less error" is one step removed from "better downstream." Freezing the backbone means visibility quality is capped by the HaMeR/WiLoR pose prior; how a stronger HPE model would do goes untested.