CoDA: A Human-LM Collaborative Codebook Development Assistant for Domain-Specific Text Annotation
Jaspreet Ranjit, Swaminathan Chellappa, Augusto Rivas Costante, Chandan Manjunath, Laura Onasch-Vera, Hailey Winetrobe Nadel, Eric Rice, Swabha Swayamdipta
cs.CL, cs.AI
2026-10-17
On homelessness notes, CoDA built a codebook from 115 reviews in two hours; Macro F1 beat a weeks-long expert codebook (593 labels) on three LMs, at one-fifth the review.
Case workers and conservation groups mostly have case notes and social posts, not filled-in tables. The variables they want are narrow, and the boundaries sit in domain language. A health referral for a young person experiencing homelessness includes arranging a ride to a clinic or help booking an appointment, as well as a handoff to a medical provider. A language model can classify that text once it has a codebook: a written definition of the concept and the decision rules. Most LM annotation systems assume the codebook already exists and spend their effort on the classifier.
Writing the codebook is the expensive step. On the homelessness task, two experts followed a standard qualitative-coding loop, labeled 593 notes over several weeks, and reached a Krippendorff's α of 0.8. Model mistakes then split in two. A scoring error means the model did not follow the rule. A conceptualization error means the rule never captured the concept the expert is trying to measure. A stronger model can only cut the first kind. The second kind means editing the codebook, and the edits have to target mistakes models actually make. Guidelines written for human readers transfer poorly onto LMs for that reason.
CoDA is an open-source system from the University of Southern California, My Friend's Place in Los Angeles, and the School of Social Work at the University of North Carolina. It runs as a web app or on a local machine. Notes can stay inside the organization, with the model served through Ollama.
Each task is one variable, cast as multi-class classification. The user uploads a task description and a label set, optionally a seed guideline, a human-labeled validation set Dval, an unlabeled corpus Dall, and a review budget b. Both case studies used a budget of 150 examples, five per iteration. The model inside the loop was Qwen1.5-32B-Chat at Q40 quantization.
One correction in the interface is typical. A note that only mentioned headaches and a request for clinic advice was predicted as referral received. The expert added a rule: if health was only discussed, the label is health discussion.
Three codebooks are compared. The baseline has only the task definition and the label descriptions. The expert codebook was written by hand over several weeks. The CoDA codebook is the iteration with the highest Macro F1 on Dval. Development used Qwen1.5-32B. Scoring used Llama-3-70B, Qwen2.5-32B, and Qwen2.5-14B, which tests whether the rules stay useful off the model that drafted them.
The homelessness task uses 6,725 case notes from My Friend's Place. Labels are physical-health referral received, health discussion, and none. All 593 expert labels are reused as Dval. CoDA ran on a disjoint pool. Macro F1 peaked at iteration 23, after 115 reviews, in about two hours, roughly one-fifth of the 593 labels in the manual process. Those two hours do not include labeling Dval.
| Model | Label text only | CoDA iter. 23 | Expert codebook |
| Llama-3-70B | 0.236 | 0.418 | 0.334 |
| Qwen2.5-32B | 0.343 | 0.493 | 0.446 |
| Qwen2.5-14B | 0.200 | 0.410 | 0.137 |
The wildlife task treats sentiment toward pangolins (positive, negative, neutral) as a proxy for trafficking. Posts hide behind nicknames and emoji. Calling pangolin soup delicious is positive sentiment and also consumption of a protected animal. Dval has 585 posts. The best codebook was iteration 20, after 100 reviews, in about one hour.
| Model | Label text only | CoDA iter. 20 | Expert codebook |
| Llama-3-70B | 0.388 | 0.556 | 0.547 |
| Qwen2.5-32B | 0.398 | 0.552 | 0.529 |
| Qwen2.5-14B | 0.429 | 0.560 | 0.519 |
CoDA beats the expert codebook in all six cells, and beats the label-only baseline in all six. On pangolin posts the margin is 0.009 to 0.041. On homelessness notes it is larger. With Qwen2.5-14B the expert codebook falls to 0.137, under the 0.200 baseline, while CoDA holds 0.410. Rules written for people can hurt a smaller model.
Absolute scores stay modest. The best cell is 0.493. Full-corpus counts are a review queue. Qwen2.5-32B flagged 1,474 health discussions and 936 referrals in the 6,725 notes, 2,410 combined. Llama-3-70B flagged 10,196 negative posts among 237K pangolin posts.
The user is usually a data lead inside a service organization. They need guidelines a model will follow, and they need the notes to stay on their own machines. CoDA puts the miss, the one-sentence correction, and the merged rule on one screen. Swapping a local model for a hosted one does not change that workflow.
The codebook travels across models. Written with a quantized Qwen1.5-32B, it still helps Llama-3-70B and both Qwen2.5 sizes. An organization can draft the rules with an open model it already runs, then switch models for the full pass.
This is an EMNLP 2026 system demonstration, built on the same group's 2026 algorithm. What it adds is an editable codebook, feedback on the label, the span, and the reasoning, and a local install. The saving is weeks of guideline writing. Classification itself still sits near 0.5 Macro F1.
Both tasks are single-target, three-class problems. Hierarchical labels, multi-label tasks, long documents, contested concepts, and several experts editing one codebook were not tried. Sampling, budgets, and stopping rules were not ablated. The loop does not check whether later notes are still forcing new boundaries.
The published CoDA number is the best iteration on Dval. The interface shows Macro F1 every round, so picking the peak is a normal way to use the tool. There is no separate test set: the same notes choose the codebook and supply the score. The homelessness expert codebook was also revised while those 593 notes were labeled, so scoring it there gives it the benefit of its own development sample. Two hours and a cut to about one-fifth the reviews both leave out the cost of building Dval. The paper says the time saved will vary with the task.
The 2,410 notes and 10,196 posts are model counts, with no audited precision on the full corpus. Negative sentiment is a trafficking lead, and it is not evidence of a sale. On the homelessness task the person who ran CoDA is a co-author. Whether an outside practitioner finishes in two hours is untested. The ethics statement is direct: these labels support expert review and are not meant to drive clinical, legal, or policy decisions alone.