Chinese-Jev: Bringing System One Model to Chinese-Language Tasks
Zexiao Wang, Zihao Zhang, Xudong Wang, Pan Wang, Ziyi Ye, Haoyu Zhao, Zuxuan Wu, Shuicheng Yan
cs.CL, cs.CV
2026-09-29
Chinese-Jev, a 322M encoder trained on 10M Chinese decisions, hits 69.20% general accuracy (1.24% above Jev) at 14 ms, 87.65% in medicine, and ~1 s on-device.
Plenty of product paths need a bounded output, not a paragraph: which service should get this request, whether a passage is relevant, which option to pick. An LLM can take those jobs as instructions. Calling a large generator for every decision is still slow and expensive.
Jev packages this into three typed decisions: choice among candidates, noul for whether a proposition holds, and score for an ordered rating, each with probabilities. The hosted API and open copies such as Open-Jev, Laya, and SemIF already run. Chinese accuracy is the weak point. Chinese labels also do not share a format: class names, yes/no, ordinal grades, multi-answer questions. A cheap System One model for Chinese is the gap, a fast decision head that does not write open-ended text.
Heterogeneous annotations are converted into one supervision format. A small encoder then scores candidates.
Each example becomes a context, an instruction, a decision type, a candidate list, and a target distribution. Single-answer and class labels become choice, with probability 1 on the annotated option. Yes/no items become noul over false/true. Discrete ratings become one-hot score targets; continuous ratings such as USTS with 19 raters spread mass between adjacent integer bins. Multi-answer questions split into option-wise noul: is this option in the answer set. Dedup hashes a normalized fingerprint, with choice candidates sorted first so permutations collapse. Decisions from the same source document stay in the same partition.
The general corpus holds 10 million examples across eight task categories, with choice, noul, and score at roughly 1:1:1. Semantic matching is 24.07% of that mix; review rating is 19.44%. Another 10,700 rule-based checks are added. Medical fine-tuning uses 3 million decisions; legal and finance use 24,000 each. CJ-Bench holds out 307,900 decisions: 100,000 general, 200,000 medical, 4,300 legal, 3,600 finance.
Fudan, NUS, and UCAS start from Laya Multilingual. The backbone is mmBERT-base, 22 layers and hidden size 768, plus two decision Transformer layers and a shared MLP, about 322 million parameters. Type, instruction, all candidates, and context go into one sequence with a marker before each candidate, so one forward pass produces all scores. An encoder-only stack is used because every candidate can be scored together, with no autoregressive decoding. A type embedding is added at every position, and the same weights serve choice, noul, and score. Choice takes argmax, noul reads ptrue, score reads the expected level. The loss is cross-entropy plus RLCD: Gaussian noise on the logits, a policy-gradient term for agreement with the target, and ranked probability score on ordered tasks. Full-parameter training runs on eight H200 GPUs, one epoch on the general mix, four epochs per specialist, four logit perturbations per example.
After Chinese general training, accuracy moves from Laya's 40.22% to 69.20%, ECE from 23.04% to 3.78%, and latency stays at 14 ms. Versus hosted Jev at 68.35%, that is +0.85 points, a 1.24% relative gain. ECE drops from 11.45% to 3.78%, a 67.0% cut. Qwen3.5-2B sits at 32.95% and 280 ms. The 20.3x figure against the Jev API includes network round-trip.
| Method | ACC (%) | ECE (%) | Latency (ms) |
| Qwen3.5-2B | 32.95 | 43.68 | 280 |
| Open-Jev | 52.33 | 10.54 | 60 |
| Laya Multilingual | 40.22 | 23.04 | 14 |
| Jev (incl. network) | 68.35 | 11.45 | 284 |
| Chinese-Jev General | 69.20 | 3.78 | 14 |
Specialists add 47.88, 21.34, and 29.33 points over the general model on medical, legal, and finance, lifting the unweighted average from 37.97% to 70.82%. Medical reaches 87.65%, 3.36 points above Jev's 84.29% (4.0% relative). Legal 60.10% and finance 64.72% still trail Jev at 68.35% and 78.39%. Average specialist accuracy is 92.0% of Jev's, at a median 15 ms, about 17x versus the Jev API and 17.5-29.2x versus Qwen3.5-2B on the same local clock.
Calibration does not track accuracy. Medical ECE falls from 17.33% to 7.25%, finance from 14.43% to 11.55%, legal rises from 15.37% to 17.45%. Mean specialist ECE is 12.08% against Jev's 5.57%. On medical, Open-Jev has the lowest ECE at 2.88%. On finance, the general Chinese-Jev checkpoint sits below the Laya init, 35.39% vs 38.31%.
An INT8 build in the iPhone 15 Pro browser on iOS 26.7 runs at about 1 second per decision on device.
Chinese routing, relevance, tool choice, and graded ratings can skip a generative LLM if the candidates are known. A 322M encoder at 14-15 ms, quantizable to a phone, speaking Jev's three decision types, is a practical head.
The general-set win is 0.85 points. Legal and finance still lag. Calibration after specialization is uneven. Medicine beats Jev with 3 million fine-tune examples; law and finance have 24,000. The backbone and RLCD come from Laya. The increment is Chinese data and domain fine-tunes.
Useful for a low-latency Chinese decision head that should not ship raw text to a cloud API. Not ready as a legal or financial adjudicator.
The paper flags remaining accuracy gaps in law and finance, and uneven calibration after specialization.
Jev latency includes API round-trip, so 20.3x and 17x are end-to-end, not same-GPU forward time. Versus Laya, latency was already 14-15 ms. Legal and finance training sets are two orders of magnitude smaller than medical, and CJ-Bench is similarly skewed (200,000 vs 4,300), so the domain average is pulled by medicine. No extra calibration is applied on the target sets, yet ECE is a headline metric. Soft-target examples are dropped from ACC and ECE.
The recipe is Laya plus Chinese data and specialist fine-tunes. There is no ablation of RLCD or the decision Transformer, and no same-size Chinese encoder baseline. Multi-answer items are scored option by option, so joint constraints are not tested. The 1-second mobile number is browser CPU, not the 15 ms GPU number, and INT8 accuracy drop is unreported.