Evaluating Multimodal LLMs as Generalist Vision-Language-Action Agents for Drone Control: Commanding, Approaching, Tracking and Searching
Jaewoo Park, Minyoung Lee, Sukmin Seo, Moonbin Yim, Hyunwook Yoon, Dohoon Ryu, Daehee Kim, Myungseo Song, Jihyuk Byun, Seunggyu Chang, Taeho Kil, Jiseob Kim, Bado Lee, Geewook Kim
cs.RO, cs.AI
2026-09-01
NAVER puts an MLLM in a drone loop with actions declared only in the prompt. Best approaching success is 65%; Qwen3.5-9B enters the 5 m radius 90% of episodes but converts 35%.
Systems that fly drones with multimodal LLMs keep shrinking what the model is allowed to decide. TypeFly parks the model outside the loop as a planner over detections. See, Point, Fly reduces control to pointing at a pixel. Fly0 leaves semantic grounding to the model and hands the trajectory to a LiDAR planner. OnFly inserts a verifier before the actuators. Fly0 itself found GPT-5, Gemini 3 Pro, Claude 3.7 Sonnet and Qwen2.5-VL-32B within 1.1 points of each other on grounding, 70.4% to 71.5%. The models look interchangeable because the hard decisions have been engineered out.
NAVER Cloud reverses the experiment. The scaffold stays fixed; the model is the independent variable. The action space lives only in the prompt: no fine-tuning, no function-calling schema. The model chooses where to fly, whether to yaw in place, whether to spend a step thinking, and when to say it has arrived. Arrival is judged after the fact from ground-truth distance. The model never sees that ruler.
DroneCATS-Agent takes an egocentric RGB frame, the instruction and the last five actions, and returns one JSON object. Four actions:
The only sensor is monocular RGB. Depth is a step-size proposal, not a range reading. One success rule covers the whole grid: at least one declaration while within 5 m of the target centre in 3D and the target is visible. Passing through the radius without declaring is a failure. Of 337 never-declaring failures, 61 had entered the radius; crediting proximity would have inflated 182 successes by a third.
The single-drone suite is 80 episodes, 20 each of approaching, searching, tracking and search-and-track, split across a residential map and a campus. Commanding adds 20 episodes in which one context flies four drones toward look-alikes that only resolve at close range. Nine models, from GPT-5 and Gemini 3.7 Flash down to Qwen3.5-2B and Cosmos3-Edge-2B. AirSim, 300 s cap.
The easiest cell is unsolved. Gemini 3.7 Flash succeeds on 13 of 20 approaching episodes (65%). Withholding the target from the first frame drops it to 40%, a ceiling no model beats. Rankings are unstable: GPT-5 approaches at 60% and tracks at 15%; Gemini tracks at 80%.
| Model | Approach SR | Approach OSR | Command N=4 SR |
| Gemini 3.7 Flash | 65 | 70 | 80 |
| GPT-5 | 60 | 65 | 20 |
| Qwen3.5-9B | 35 | 90 | 15 |
| Qwen3.5-2B | 0 | 30 | 5 |
| Cosmos3-Edge-2B | 0 | 25 | 0 |
OSR counts any trajectory that entered the 5 m radius, declaration ignored. Qwen3.5-9B enters on 90% of approaching episodes, more often than any frontier model, and converts 35%, declaring on average at 0.63 of the start distance. Qwen3.5-2B declares at 1.28 of the start distance and never succeeds. Cosmos3-Edge-2B enters the radius 25% of the time and never declares.
Commanding four drones from one context widens the protocol split. On steps where Qwen3.5-9B commands go for all four, it emits the same point 70% of the time (27B: 58%). GPT-5, Claude Opus 5 and Gemini Robotics-ER 2 never do this. Gemini 3.7 Flash goes from 65% single-drone to 80% commanding; GPT-5 drops from 60% to 20%. The think action was invoked 1,797 times; step-level observation does not show it systematically changing outcomes.
This is a cleaner diagnosis for onboard deployment. Spatial perception and closing distance already work at 2B to 9B for a large share of episodes. What fails is holding a prompt-declared action protocol across dozens of steps, especially the terminating finished action. Prior systems decide termination outside the model, which is a plausible reason this gap never shows up in their numbers. If the budget is edge compute, train when to stop and how to address distinct views, not another pixel pointer.
The benchmark is usable as-is: one success rule, a swappable model. Control rate in the paper is about 2 Hz on datacentre hardware; the authors note that host slowdown, especially on moving targets, pulls success down with it.
Simulation only, no hardware. Twenty episodes per cell imply binomial noise of about 9 to 11 points; three Gemini flights total 43.7±7.8 of 80, so neighbour rankings inside a tier should not be over-read. Commanding is not a matched control: it adds close-range disambiguation and swaps the campus map for Blocks. Hierarchical commanding is unevaluated. think has no isolated ablation. Distance is to the target centre; large objects would read differently at the surface.