GKDT: General Keypoint Detection Transformer
Changsheng Lu, Yuxin Chen, Haokun Gui, Rong Wang, Jie Yang, Harry Yang, Anton van den Hengel, Jiaya Jia
ECCV 2026
cs.CV
2026-07-01
GKDT unifies 1.3M instances from 29 datasets into one keypoint-detection model; on 22 test sets it clears 90% [email protected] on 17, approaching dedicated expert models.
Keypoint detection, the task of placing semantically meaningful points on an object (joints on a person, eyes on an animal, corners on a part), has long been a one-model-per-category affair. Human pose has its own networks, and so do faces, hands, animals, garments, and X-rays. The cost is predictable: every new species or object class needs fresh annotation and a fresh training run, and nothing transfers. A model trained on human anatomy has nothing useful to say about a fish or a sofa.
The paper asks a single question: can one model, driven by a prompt (a sample image, a text description, or both), detect keypoints on any object, including classes it never saw in training?
GKDT is built on the DINOv3 vision foundation model. The centerpiece is a kernel-generation transformer, motivated by a simple observation: in specialist models the learned convolution weights act as keypoint prototypes, matched against image features to localize points. The authors flip this around. If you can learn such prototypes from a prompt, you no longer need per-category weights.
The pipeline has four steps. It extracts visual keypoint prototypes from Gaussian heatmaps on a support image and text prototypes from written keypoint descriptions. A kernel-generation transformer then uses self- and cross-attention to turn both sets of prototypes into convolution kernels. Those kernels convolve over the query image's features to produce heatmaps directly. Detection is non-parametric; no category-specific weights are learned.
Two training tricks carry the generalization. Mix-modal prompted training randomly picks visual, text, or both as the prompt each episode, mimicking how real users prompt inconsistently. Dynamic importance sampling handles the long tail: MegaKPT's head classes (humans, animals) dwarf the tail, so during sampling the method checks whether the drawn class is a head class and, if so, removes the sample instead of replacing it, gradually flattening the distribution without dropping a single tail example.
The paper releases MegaKPT alongside the model: 29 existing datasets unified into COCO format, with 935,343 images, 1,348,228 instances, 1,587 categories, and 740 keypoint types spanning humans, faces, animals, insects, furniture, vehicles, clothing, and medical scans. It is the first dataset to ship expert-written keypoint text descriptions for medical images (cephalometric and hand X-rays).
On 22 test sets, 17 clear 90% [email protected] under dual prompting. Against baselines DINOv3, CapeFormer, OpenKD, and X-Pose:
| Test set | OpenKD | GKDT |
| Vinegar fly (dual) | 95.97 | 98.79 |
| Hand X-ray (zero-shot) | 85.31 | 99.60 |
| Cephalometric | 99.18 | 99.49 |
On COCO multi-person pose, the larger GKDT-H variant reaches 78.1 AP, within striking distance of the dedicated ViTPose-H at 79.1. One general model nearly matches an expert trained for a single task. Training took 16 H800 GPUs for 20 epochs.
Keypoint detection is the shared substrate under pose estimation, action recognition, and medical image analysis, and most teams still build bespoke detectors for their own object classes. GKDT shows a workable route to covering a long tail of categories with one promptable model, which matters most where labeled data is scarce and time to deployment is short. MegaKPT fills a gap of its own: until now there was no unified benchmark this broad that also carried text descriptions.
The paper has no dedicated limitations section. A few issues surface in the text. The multi-object case is two-stage: an external detector (such as Grounding DINO) produces boxes first, then each box is processed, so performance is bounded by detector quality, and the authors report noticeably different numbers depending on the boxes used. The "17 over 90%" figure is for dual-prompt mode; text-only prompting is clearly weaker on several non-medical sets. Training needs 16 H800s, which is not lightweight for small teams. Discussion of failure cases and the effect of backbone size is thin.