Gricea: An Open Science Platform for Conversational AI Research
Nikhil Sharma, Yunlin Gong, Xinyang Cheng, Ziang Xiao
cs.HC, cs.AI
2026-09-19
Gricea encodes CAI studies as Study Flow and Task Flow graphs. It rebuilt 93% of eligible CUI 2026 configs, flagged gaps in 96%, and let 10 researchers author runnable experiments.
A controlled study of conversational AI has to hold together procedure, interface, model settings, retrieval context, and logging. Those pieces usually live in a survey tool, a one-off frontend, a prompt file, and a pile of scripts. The paper states the hypothesis. The next lab cannot recover which UI participants saw or which system prompt the agent used. Commercial chat products also change models and policies, so the same prompt next year is not the same condition.
Gricea's claim is that the study design should itself be a runnable, inspectable, forkable artifact, not a paragraph in the appendix.
A formative pass collected 100 candidates on conversational agents and user studies, then kept 57 papers that centered a participant-facing system and described the design. Coding produced six coupled dimensions: interface, agent, context and grounding, task and modality, domain and audience, and study procedure. Five requirements follow: make the condition explicit, keep procedure and task distinct but linked, cut engineering cost, keep the artifact inspectable and reusable, and stay extensible.
The runtime is two directed graphs. Study Flow routes people through consent, pre-task measures, assignment, task entry, post-task measures, branching, and randomization. Task Flow sits inside a task node and runs that interaction: model calls, retrieval, voice, loops, interface updates. Variables bind assignment into the task config, so what varies and what stays fixed live in one definition. Researchers edit both graphs on a no-code canvas, preview the participant view, and publish a frozen Study Version of graphs, prompts, materials, and bindings. The runtime executes that version. Later edits open a new draft and cannot rewrite a condition already in the field.
The participant runtime covers chat, search, voice, coding, annotation, image generation, and split views. Instrumentation can record timestamps, clicks, focus, scroll, and typing, plus optional voice, masked replay, and eye tracking. The community layer shares full studies, node templates, RAG document sets, and de-identified data.
The replication corpus is CUI 2026 full papers, held out from the formative set. Of 37 papers, 29 report participant-facing human studies. Gricea could represent and run the procedure, interface, or conversational task for 27 of 29: 10 complete, 17 partial, matching the abstract's 93%. Two studies stopped on missing interview and survey materials. Twenty-eight of 29 papers, 96%, omit something: questionnaires or interview wording in 21, full agent prompts or grounding rules in 15, stimulus materials in 11. Fourteen papers need robots, wearables, voice cloning, or in-person coordination; those stayed as external dependencies and were only partially rebuilt.
The authoring study recruited 10 people: four PhDs, two master's students, two professionals, one professor, one undergraduate, five women and five men, median age 25–34. Sessions lasted 90 minutes and paid $30. After a guided task on how an assistant's stance affects trust, each person implemented an open research question. All ten studies ran, covering tool-call reliance, serendipity in search, proactive versus reactive coding agents, the "AI penalty" in evaluation, chatbot personas in mechanical design, a moderated two-agent voice debate, fact-checking, and model-tier effects on the digital divide.
| Measure | Mean | SE |
| Would recommend | 6.8 | 0.13 |
| Ease of use | 6.1 | 0.31 |
| Confidence others can reproduce | 5.9 | 0.37 |
| Could express the intended design | 5.5 | 0.40 |
P8 estimated "at least 3 to 6 months" and about $25,000 saved. P9 does not code and had assumed this kind of study required paying someone to build it. Friction clustered on variables and branching, and on dropping from Study Flow into the finer Task Flow. Some named a steep learning curve; some asked for an assistant that proposes an initial template from a research question.
Labs running conversational studies usually need a record that binds procedure to task and can be reopened after publication, not another chat widget. Gricea turns reproducibility from a methods paragraph into a version that runs. The CUI numbers show that peer-reviewed papers still drop prompts and questionnaire text; a methods section is not enough. The no-code canvas is a real drop in barrier for people who do not ship systems. For people who already write code, the value is frozen versions and forks, not the end of engineering.
Ten authors do not represent HCI or the social sciences. Replication covers CUI 2026 only. Participatory workshops, in-the-wild device use, and the physical behavior of embodied robots were out of scope for the reconstructions. The platform can express a wider design space than this evaluation used. A 6.8 recommendation score comes from a post-tutorial questionnaire and carries novelty bias. P8's time and money figures are self-reports, not a controlled comparison. The learning cost of branches and variables is in the results; people without a clear research question are told the tool works best once the RQ exists.