Dev reproduces zero-shot classifier benchmark to 4 decimals, loses to keyword rules

JhouHate · reddit · 2026-09-25

A developer ran a two-step validation of Laya (convaiinnovations/laya), an open-weights zero-shot classifier that runs locally:

Step 1: verify the tool. Reproduced the published MASSIVE (en) score of 78.33% to the 4th decimal across ROCm GPU, Windows CPU, and Linux CPU setups.

Step 2: test on real data against three baselines — majority class, a keyword if/else rule written in an afternoon, and TF-IDF + logistic regression:

| Task | Majority | Keyword | TF-IDF+LR | Laya |

|---|---|---|---|---|

| Paper sections, 5 classes (n=5,489) | 42.4% | 35.5% | 70.5% | 25.6% |

| Same, balanced | 20.0% | 18.1% | 45.7% | 18.9% |

| Work-note type, 3 classes (PT) | 33.3% | 47.4% | 80.7% | 47.4% |

| Work-note area, 16 classes (n=354) | 34.2% | 32.5% | 46.1% | 27.4% |

On a balanced yes/no task (n=242) it scored 49.6% with 97.5% mean confidence. The author acknowledges the comparison isn't fully fair (TF-IDF saw labeled data, Laya didn't) and suspects the real bottleneck is short utterances vs long passages rather than language. The model seems strongest where its benchmark lives: short sentences, broad categories, English. The full pipeline is packaged as a reusable kit (GitHub: JhouCode/laya-fit-check), and the author has raised the issue in a Hugging Face discussion.

Original post →

More from Research

• Grady Booch: LLMs Only Resemble the Brain at Its Most Primitive Structures — Grady_Booch · 2026-09-25
• ICLR 2027 Submission De-anonymization Incident Sparks OpenReview Statement — Striking-Warning9533 · 2026-09-25
• FineVision, an open-source 24.3M-sample VLM training dataset, accepted to NeurIPS — andimarafioti · 2026-09-25
• LLM Verbalized Confidence Paper Accepted at EMNLP UncertaiNLP Workshop in Budapest — yeewhye · 2026-09-25
• Valerio Capraro joins JBEE editorial board to handle AI and behavioral economics papers — ValerioCapraro · 2026-09-25
• ETH Zurich and Anthropic built an AI that doxes anonymous accounts in a minute for $2 — skdh · 2026-09-25

Research channel →