A Common Measure of Communication for Speech Brain-Computer Interfaces
Dulhan Jayalath, Benjamin Ballyk, Oiwi Parker Jones
cs.LG, q-bio.NC
2026-09-03
OVMI weights in-vocab mutual information by lexical coverage so speech BCIs share one scale; maximising it for vocab selection lifts accuracy up to 16.3% over frequency.
Speech brain-computer interfaces map neural activity to words. Studies differ in recording hardware (intracortical arrays, MEG, EEG, fMRI), in task (attempted speech versus perceived speech), and in vocabulary size, from 50 words to 125,000. Accuracy and word error rate are computed only over the words a given system supports. Ninety percent on a 50-way task and ninety percent on a 1,000-way task are not the same quantity of information, so published scores rarely compare.
Shared-dataset benchmarks can rank methods inside one experimental setting. They cannot put an implanted attempted-speech decoder and a non-invasive listening decoder on the same neural test set. Forcing a universal vocabulary is equally awkward: the same words occur at very different rates across corpora, or not at all. Oxford's Neural Processing Lab splits the measurement problem into two questions: what distribution of words a user may want to say, and how much information from that distribution a decoder actually conveys. The proposed answer is open-vocabulary mutual information (OVMI), computed against an explicit reference distribution p.
OVMI multiplies in-vocabulary decoding fidelity by the chance that an intended word is in the vocabulary at all.
Lexical coverage C(S) is the probability mass of the reference distribution p that lands inside vocabulary S. In-vocabulary mutual information I(X;Y | X∈S) is how much seeing the decoder output cuts uncertainty about the intended word, given that the word is supported. OVMI = C(S) × I(X;Y | X∈S). The full mutual-information split also contains a binary entropy term H2(C(S)), which only says whether the intended word is in-vocab. The paper drops that term so "detecting an out-of-vocab word" is not counted as communication.
A toy ranking reversal makes the difference concrete. A user draws uniformly from 1,000 words. System A supports 50 words at 100% in-vocab accuracy, 5% coverage, 0.28 bits of OVMI. System B supports all 1,000 words at 50% accuracy, 100% coverage, 3.98 bits. Accuracy ranks A first. OVMI ranks B first.
Most published speech BCI papers report a scalar accuracy or WER, not a confusion matrix. The authors therefore use a Wolpaw-style symmetric-error approximation: every supported word is correct with probability P, and errors spread uniformly over the remaining V−1 words. P is macro accuracy, so evaluation-set frequencies are not counted twice; frequency already enters through p. When a reliable confusion matrix exists, OVMI should be computed from the empirical channel. When C(S)=1, OVMI reduces to ordinary in-vocab mutual information. Add a uniform prior and symmetric errors and it recovers Wolpaw's per-trial information; multiply by trial rate for an open-vocabulary information transfer rate.
The default reference is SUBTLEX-UK (subtitle-derived spoken English), with entropy 9.77 bits. Three further references are Switchboard conversation (8.27 bits), the 36-word Universal Core Vocabulary for AAC (4.30 bits), and a Sherlock Holmes narrative chapter (8.44 bits). Cross-domain comparisons report OVMI divided by H(p).
For a noiseless decoder, Wolpaw's log2 V, in-vocab entropy H(pS), and OVMI's C(S)H(pS) diverge sharply. At small V the coverage gap dominates: a perfect decoder cannot convey a word it does not support. As V grows, coverage approaches 1, yet the uniform-prior gap keeps widening because word frequencies are Zipfian. Invasive and recent non-invasive systems typically use 50–250 Zipfian words, where both sources of overstatement are large.
Re-scoring published systems against SUBTLEX-UK:
| System | Vocab | P | OVMI / H(p) |
| Card (+LM) | 125k | 97.5% | 93.7% |
| Willett (+LM) | 125k | 76.2% | 72.0% |
| Willett isolated | 50 | 94.0% | 6.7% |
| Moses (+LM) | 50 | 74.4% | 4.7% |
| Tang fMRI | 6867 | 6.7% | 3.6% |
| LibriBrain100 | 50 | 25.8% | 2.4% |
| MEG-MASC | 50 | 8.5% | 0.3% |
The historical jump among implants comes from growing the vocabulary from 50 to 125k after in-vocab fidelity had already saturated. Non-invasive word decoding stays in a low-information regime: LibriBrain100 reaches 25.8% balanced top-1 on 50 words; MEG-MASC sits at 8.5%. For perceived speech, OVMI should be read as lexical decoding capability relative to the same speech distribution, not as a usable communication interface.
Small-vocab rankings move when p changes. The 50-word Willett system (+LM) conveys 6.4% of broad spoken English and 40.4% of the AAC reference; Moses moves from 4.7% to 30.7%. LibriBrain100 and isolated-word Moses are nearly tied on broad speech (2.43% vs 2.4%), Moses leads on AAC (16.5% vs 7.34%), and LibriBrain100 leads on narrative (2.62% vs 1.0%). Large-vocab implants are comparatively insensitive: Card stays in 93.7–97.5% across four references, Willett 125k in 72.0–76.2%.
OVMI also works as a vocabulary-selection objective. On the d’Ascoli et al. (2025) contrastive decoder, a 250-word candidate pool is trained once and subsets are chosen at inference. Against frequency selection, OVMI's peak relative accuracy gains are 15.4%, 16.3%, and 8.4% on TIMIT, podcasts, and Sherlock. The edge shrinks as V grows; at V=250 the candidate pool is exhausted and methods coincide.
Speech BCI progress is currently hard to read across labs. Controlled benchmarks still answer which method wins inside one setting. OVMI answers what that capability is worth against an explicit communication target. Benchmarks need shared neural data. OVMI needs a vocabulary, an accuracy or WER, and a public word-frequency table.
When the supported words are a design choice, selecting them to maximise OVMI beats taking the V most frequent types, because frequent words are not always the ones the decoder can tell apart. Clinical reports should include an AAC reference; scoring a 50-word caregiving vocabulary only against unrestricted English systematically underrates it. Large-vocab implants have largely closed the coverage gap. Non-invasive systems are limited by signal fidelity outside the skull; growing the vocabulary will not rescue MEG-MASC's 0.3%.
This is a measurement paper. It does not train a new decoder, and a higher OVMI does not mean a more practical clinical paradigm.
Retrospective scores rest on the scalar estimator. Symmetric errors and a single P are strong assumptions: real confusions cluster among phonetically similar words, so OVMI could be biased either way. Continuous-speech P is set to 1−WER, a lower bound, because insertions inflate WER without missing an intended word. Card's 97.5% and Willett 125k's 76.2% carry that dagger.
Out-of-vocabulary words count as untransmitted. Paraphrase around a missing word is invisible to the metric. The measure is lexical unigram, not contextual; a future conditional-language extension is left open. Both caveats are stated in the discussion.
The 125k lexicons are unpublished; coverage is approximated from 126,052 unique spellings in the CMU Pronouncing Dictionary, while the channel still uses reported V=125,000. Perceived speech (listening to stories, watching films) is not attempted speech. The authors are explicit that a higher OVMI does not by itself imply a usable interface.