Industrial-Instruction: An End-to-End Framework for Building Instruction-Tuning and Benchmark Datasets from Industrial Technical Reports
Parsa Bakhtiari, Hassan Bashiri, Alireza Khalilipour, Masoud Nasiripour, Moharram Challenger
cs.CL
2026-08-24
906 Panasonic PDFs become ~13.6k industrial QA pairs. Full FT of Qwen3-4B lifts Set-Match from 28.5% to 42%; LoRA barely moves, paraphrased FailureSensorIQ stays at 0%.
The knowledge that actually runs a plant often lives in product manuals, spec sheets, and failure tables: prose mixed with tables, scattered across PDFs. Generic RAG and generic instruction tuning barely touch that distribution. FailureSensorIQ already showed how hard industrial multiple choice is: large models average about 53.5%, and open models tuned on general QA average about 29%. There is no public instruction-tuning or benchmark set built from real industrial technical reports.
This work, from Hamedan University of Technology and the University of Antwerp, ships two downloadable industrial QA datasets and the PDF-to-dataset pipeline behind them. Models under 10B are the intended consumers. On a factory floor, deployability beats calling the largest API.
The source corpus is 906 public Panasonic PDFs (7,525 pages). Layout extraction uses 3B Dots.OCR, keeps Markdown tables, drops every image, then removes 370 empty, tiny, or duplicate pages. 291 of 7,525 pages fail extraction (about 3.8%); tables account for 236 of those failures. Retrieval is EmbeddingGemma (300M) plus FAISS.
Questions are synthesized under five query–document relations, matching the dirty cases real RAG hits:
FailureSensorIQ's 5,334 items are used as simulated instructions, forcing five-way (possibly multi-label) JSON. The same retrieval and prompts are run twice: open-weight Qwen3-30B-A3B-Instruct, and Claude-Opus-4.6 via API. Qwen yields 23,910 raw pairs; rule filters drop about 43% (10,353), leaving 13.6k (12,557 train / 1,000 test). Claude yields 26,395 raw pairs and drops only 143 (0.5%), with 25,252 train / 1,000 test. Local generation took about 1 h 43 min and $3.2; the Claude run cost about $330.
Downstream, Qwen3-4B-Instruct is fully fine-tuned for about 12 hours on two RTX 5090s. LoRA ranks 8 through 64 barely move the needle. Metrics are Set-Match Accuracy (set equality, order ignored), F1, and Jaccard, plus FailureSensorIQ and full MMLU before and after the tune.
Before tuning, on the Panasonic split, Qwen3-4B scores 28.5% Set-Match and 46.7% F1; Phi-3-mini 17.5%; RAG-Instruct-Llama3-8B 0.7%. The best LoRA run moves Set-Match from 28.50% to 28.70%. Full fine-tuning is what actually moves it.
| Data / setting | Set-Match | F1 |
| Qwen-synthetic, base | 28.5% | 46.6% |
| Qwen-synthetic, full FT | 42.0% | 63.5% |
| Claude-synthetic, base (with RAG) | 40.9% | 58.6% |
| Claude-synthetic, full FT (with RAG) | 56.4% | 72.7% |
Each test split is drawn from its own generator, so the baselines are not on one ruler. The direction is consistent: full FT adds about 13 to 15 points on the Panasonic distribution, and the simplest RAG adds almost nothing once the model has seen that distribution.
On FailureSensorIQ the story splits. Qwen-data FT drops AccOrgIBM from 34% to 27% and lifts F1-Micro from 66% to 74%. Claude-data FT lifts AccOrgIBM to 49.6% and drops F1-Macro / F1-Micro to 33.5% / 50.3%. Both tunes score 0% on the paraphrased AccPerIBM split; RAG-Instruct-Llama3-8B scores 33% there and is nearly unusable on Panasonic.
Full MMLU (14,042 items): base 72.13%, Qwen-data FT 70.87% (−1.26), Claude-data FT 72.08% (−0.05). Forgetting concentrates in Humanities, with about a 10-point drop on moral-scenario subjects for the Qwen-data run.
This is a reproducible industrial-document factory: table-preserving OCR, five dirty retrieval scenarios, an open versus closed generator comparison, and full FT of a small model. $3.2 produces trainable data; $330 buys cleaner JSON and a larger gain that does not scale with the two-order cost jump. The usable conclusions are narrower. LoRA is ineffective on this task. Naive RAG barely helps after the model has memorized the same distribution. Self-generated Panasonic scores will look good; rephrased FailureSensorIQ questions send the 4B backbone to zero.
The corpus is public English Panasonic documents: one company, one house style. Images and drawings are dropped as a class, and in industrial pages the figure is often the spec. Questions are model-written; filters catch format, not factual correctness. The Panasonic benchmark is in-distribution with the training set, so part of the full-FT gain is fitting the generator's item style. The two test sets are not comparable. AccPerIBM is 0% throughout; the pipeline never emits paraphrased or adversarial variants. The authors list that as future work. Several tables in the PDF are misaligned, so numbers need to be checked against the surrounding prose.