arXiv News

First-hand information for everyone

Site

AboutPrivacy PolicyTerms of Use

© 2026 arXiv News

Large‑language models can generate diagnostic Q‑matrices, and their reasoning traces provide insight into potential uncertainties | arXiv News

arXiv News

EnglishJapanese

Switch language

EnglishJapanese
Loading account…

First-hand information for everyone

Back to front page
Artificial IntelligenceEnglishPublished

Large‑language models can generate diagnostic Q‑matrices, and their reasoning traces provide insight into potential uncertainties

November 22, 2026arXiv: 2611.26422v1

Researchers introduced Reasoning‑Trace Analysis (RTA), a new framework that asks large‑language models (LLMs) to produce step‑wise, attribute‑labeled reasoning (the so‑called chain‑of‑thought, or CoT, prompts). These CoT traces are then inspected to reconstruct the Q‑matrices that define which attributes each test item requires. The method is designed to help experts check that AI‑generated matrices reflect intended skill sets.

In a simulation study of ten fraction–subtraction items with added noise, four different LLMs—DeepSeek v4 Flash, Claude Haiku 4.5, Kimi K2.6, and GPT‑4o mini—successfully recovered the full target attribute vectors for all items. Three models achieved perfect overlap (1.00) with the intended matrices, while GPT‑4o mini had an overlap of 0.80, missing some attributes because it mislabeled operations.

The same four models were tested on 11 real fraction‑subtraction items answered by 536 examinees. They generated Q‑matrices under three temperature settings, with 100 replications each. Q‑matrices produced by direct generation (without CoT) overlapped the reference matrix on average at 0.877, whereas CoT‑derived matrices overlapped at 0.809. The direct approach was also more stable: the dominant‑matrix proportion—a measure of consistency across replications—was higher than for the CoT workflow.

When inspecting the CoT traces, researchers found that differences in operation order did not affect the final attribute set but caused trace‑analysis variability. Mislabeling of combined steps—e.g., merging a borrowing operation with a whole‑number separation—led GPT‑4o mini to omit key attributes on some perturbed items. These trace‑level variations also showed that different temperature settings and model families produce diverse Q‑matrices, reflecting sensitivity to stochastic sampling and design choices.

Overall, the study shows that LLMs can approximate expert‑specified Q‑matrices, but evaluating the reasoning traces is essential to identify when and why they fail. Future work will refine attribute labeling rules and explore how to combine RTA with tighter expert review to ensure reliable measurement tools.

Source paper

Reasoning-Trace Analysis Using Large Language Models for Interpretable Q-Matrix Construction

Artificial Intelligence
View on arXiv