2026-10-24
CodedLang span-annotates coded wording in 900 of 7,744 Chinese Google reviews. Best detection F1 is 0.78, and GPT-5-mini rating error rises from 0.52 to 0.68.
Chinese-speaking users on Google Maps sometimes refuse to type "not tasty" in a bad review. Said quickly, bu4 hao3 chi1 slides toward bao4 chi1, which is written 报吃 and machine-translates as "newspaper eat." The English looks broken. Readers in the group still hear a complaint. Homophones, pinyin abbreviations, emoji, and cross-language sound-alikes all do this on purpose: the surface string is not the message, and translation or moderation is meant to miss it.
Models assume the literal string is the meaning. Phonetic and lookalike substitutions were long treated as noise to normalize, or as attacks on a filter. In real reviews they are a way of talking, and a rare one. The Penn State and University of Washington authors hand-labeled 5,391 Chinese reviews of U.S. restaurants and found 19 coded cases, 0.35%. Without span labels and a taxonomy, a miss and a misread look the same.
CodedLang keeps Chinese-script text from two public Google Maps corpora: 5,391 restaurant reviews and 112,521 local-business reviews. Mining is a human loop around a dictionary. Twenty-three examples from social media and news were span-labeled into a seed of about 100 entries, then matched across the corpus. Reviews whose machine translation still contained pinyin were pulled in too, because that usually means the translator already failed. Two authors checked every candidate, added new spans, and searched again until a round produced no new accepted code. A homophone list from Weibo and Tieba was merged in.
Of 7,744 reviews a person actually read, 900 contain coded language, across 184 unique spans. A review can hold several spans. Each span gets one primary strategy. If a phrase stacks tricks, for example a homophone that ends as a hamburger emoji, the label follows the most visible surface step. Agreement on "coded or not" is Cohen's κ = 0.99, and 98% of spans overlap. The recurring fight is dialect versus deliberate code. The Beijing expression 里格楞 was marked non-coded because the surface form is the meaning.
The seven classes are ambiguous homophones that are real words, such as 炒鸡 for 超级. Non-lexical homophones that are not words, such as 灰常 for 非常. Pinyin, zhuyin, or Latin-letter substitution. Emoji substitution. Orthographic lookalikes. Cross-lingual sound encoding, including 古德 for "good" and the newspaper-eat case. Cipher sentences with no literal reading. The counts are skewed: 333 ambiguous homophones, 292 non-lexical, 200 phonetic substitutions, 48 emoji, 47 ciphers, 34 orthographic, and 7 cross-lingual.
All three tasks ship the definition and a few examples in the prompt. Detection is binary. Classification must return classes, exact spans, and a short reason. Rating prediction compares five views of the text: ordinary reviews, coded reviews left as written, character-masked codes, span-masked codes, and codes replaced by the annotated dictionary.
No detector wins every column.
| Model | Precision | Recall | F1 |
| DeepSeek-V3.2 | 0.77 | 0.79 | 0.78 |
| Gemini-2.5-Flash | 0.67 | 0.92 | 0.77 |
| GPT-5-mini | 0.69 | 0.80 | 0.74 |
| Qwen2.5-7B-Instruct | 0.80 | 0.43 | 0.56 |
| Llama-3.1-8B-Instruct | 0.35 | 0.68 | 0.46 |
DeepSeek-V3.2 has the best F1, and it is still 0.78. Gemini's recall of 0.92 comes with precision of 0.67. Qwen2.5-7B reaches precision 0.80 and recall 0.43.
Classification is uneven, and class sizes differ by two orders of magnitude. DeepSeek-V3.2 scores F1 0.74 on ciphers and 0.40 on cross-lingual encoding, where precision is 0.25 and the class has 7 items. Homophone, phonetic, emoji, and orthographic F1 values sit between 0.42 and 0.59. GPT-5-mini reaches 0.84 on ciphers and 0.10 on cross-lingual text. Orthographic labels tend toward high recall and low precision: ordinary lookalike characters get swept in.
Star rating is mean squared error, lower better. GPT-5-mini scores 0.52 on non-coded reviews, 0.68 on coded originals, and 0.55 after dictionary decoding, close to the non-coded number. Character masking rises to 0.71. DeepSeek-V3.2 is the best of the five on non-coded text at 0.47, jumps to 0.80 on coded originals, stays at 0.82 after decoding, and hits 1.16 when the whole span is masked. Decoding does not help it. Qwen2.5-7B is worst on coded originals at 1.90. Ratings of 1 and 5 are easy. Three-star reviews are hardest. On ratings 2 to 4, masking the code beats both the original and the decoded text. The phonetic surface misleads the model, and swapping in the "real" words does not always clarify intent.
Coded reviews sit at lower stars: chi-square p < 0.001, Cramér's V = 0.29, ordinal coefficient −0.89. Phonetic distance is character error rate on pinyin and IPA. Non-lexical homophones are closest to the intended phrase, pinyin CER 0.30. Cross-lingual encoding is farthest, pinyin 1.23 and IPA 2.46. GPT-5-mini's rating error on those two classes is 0.43 and 0.86. Across the five classes, Spearman ρ between error and CER is 0.80 with p = 0.104. The direction is there. Five classes are not enough to call it.
Review ranking, translation, and moderation all assume literal sentiment is the sentiment. This is a reproducible counterexample. After the prompt has already explained the taxonomy, strong models top out near F1 0.78 on detection, mostly miss 0.6 on the fine classes, and fail cross-lingual codes. Dictionary decoding pulls GPT-5-mini from 0.68 back to 0.55, near its 0.52 on plain text. DeepSeek-V3.2 moves from 0.80 to 0.82, slightly worse. Qwen and Llama improve after decoding and remain well above their own non-coded error. "Decode, then score" is not a default that transfers. Of 184 spans, 111 appear once. The most common span, 滴 standing in for 的, appears 102 times and usually sounds cute. Codes that spread turn into memes and stop hiding anything. The ones that still hide are the one-offs, which is exactly what a dictionary loop misses.
The authors put coverage bias first. The bootstrap only finds forms that were in the seed or that the tools recognized. A new code in neither place is absent. Collection stops in 2022 and in U.S. Google reviews. Later slang, other platforms, and other regions are out of scope. Two linguistically trained native speakers did the labels. Insider jokes can still be missed or misread. High κ means those two people agreed. Seven cross-lingual examples cannot support an F1 of 0.10 or 0.40 as a general rate. Detection and classification prompts already include definitions and examples, so these scores are after the model has been shown the answer key's shape. The authors say stars and text mostly agree, with 7 reviews at 4 or 5 stars whose content is negative. The denominator of that check is not stated.