73 underwater quality metrics vs human judgment: best κ=0.23, and UCIQE favors edits people reject

Do Underwater Image Quality Metrics Agree with Human Perception? A Benchmark with Human Choices, Simple Edits and Generative Drift

Rishii Kumar Singh, Abhinav Rajput, Dinesh Kumar Vishwakarma

cs.CV, eess.IV

2026-11-14

Tests 73 underwater image quality metrics against five human reference sets. Best worst-case kappa is 0.23 (Q-Align); the widely used UCIQE prefers edits people reject.

What problem this solves

An underwater photo has no clean reference; there is no water-free version of a real reef, so enhancement papers rank methods with no-reference image quality assessment (NR-IQA) scores, most often UIQM and UCIQE. The pipeline rests on one assumption: a higher score means a better-looking image. Smaller studies had already found both metrics out of step with perceived quality, without changing the practice. This paper measures the question end to end: 73 metrics against five human reference sets, three stress tests on top, everything released as a public benchmark.

Method

The five references are four published opinion-score sets (UID2021, SAUD, SAUD2.0, SOTA; 12,360 images over 1,160 scenes) plus a new two-alternative forced-choice study: 32 raw UIEB photos each paired with one of 12 edits, 20 raters, 1,920 choices. The study fills a gap, since published sets contain only outputs of enhancement methods, never the simple edits a metric might over-reward.

Agreement is measured within a scene: for each pair of enhancements of the same photo, does the metric pick the one people scored higher, pooled into Cohen's kappa. This matches how enhancement papers actually compare methods, and it drops whole-set rank correlation, which rewards telling different photos apart. Median kappa between two raters is 0.41, the reference line.

Three stress tests:

One forensic find: the public MATLAB UCIQE code returns NaN for any image containing a pure-black pixel, which hits 27% to 38% of the opinion-score images.

Results

ItemValueReference
Best worst-case kappa of 73 metrics0.23 (Q-Align)raters agree at 0.41
UCIQE on the four opinion-score setsnegative on three: −0.08 / −0.21 / −0.170.18 on UID2021
S+S+C edit beats best human-rated enhancement under UCIQE60% / 86% / 95% of scenespeople reject the edit
Raw photo after 10 PGD steps (ε=2/255) beats human favorite150 of 150 scenes, all 5 metricsPSNR still ≥ 43.8 dB
Scores detecting drift on the Pro replica chain22 of 73 (strict rule)noise grows ×1.4 to ×2.8

The widest human-machine gaps: after 20 rounds of saturation, sharpening and CLAHE, 1% of human votes want the result while UCIQE picks it on 100% of photos; strong Gaussian noise gets 9% of votes and all of UCIQE's choices. On the seven edits people reject, UCIQE still sides with the edit in 91% of pairs. UCIQE rises on all four drift chains; only five scores detect drift on all four, and RALI is the only default checkpoint among them.

Learned underwater metrics collapse outside their home lab: the five with a known training set reach kappa 0.60 to 0.92 on it and at most 0.31 elsewhere; UIQAVSI hits 0.92 on SAUD, its training set, and at most 0.19 on the three sets from other labs. Three models never trained on underwater data (Q-Align, VisualQuality-R1, RALI) beat every learned underwater metric outside its own lab on SAUD, SAUD2.0 and the study, though not on UID2021 and SOTA.

Why it matters

For anyone building or reviewing enhancement work, this is an audit you can apply directly. A leaderboard sorted by UCIQE is hard to defend when its kappa is negative on three of four opinion-score sets. The five recommendations are concrete: report within-scene agreement on at least two references from different labs and report the lower value; never rank by UCIQE alone; include the raw photo and a simple S+S+C edit as baselines, because a method that cannot beat them has shown no gain; report metric implementations and versions; for generative enhancement, measure noise and color change directly. TrustUIQA releases the human references, attacked and drifted images, and every metric's scores on GitHub and HuggingFace, so new metrics run through the same pipeline. The drift test generalizes to any image-editing model fed back into itself. This is measurement work rather than a new method; the value is coverage and systematic stress testing.

Limitations

The authors are candid: 20 raters, all from one university, on their own screens, agreeing with each other at only 0.41; blocks of 2, 3, 9 and 6 raters leave some photos with few votes. No human directly compared a simple edit against the best enhancement, or a late drift frame against an early one, so those inferences are indirect. The drift test covers 20 photos and two editors, whose models may change between versions. Attacks hit only five general-purpose metrics, none of the underwater ones. Two more points from this reading: absolute kappa depends on the pair-order convention (the best worst-case moves from 0.23 to 0.26 under the alternative), and metric rankings are descriptive with no significance testing. VisualQuality-R1 ties on roughly 20% of pairs and ties are excluded, which flatters it.

Terms

Source

Related papers

All paper explainers