First-hand information for everyone
Researchers introduced Reasoning‑Trace Analysis (RTA), a new framework that asks large‑language models (LLMs) to produce step‑wise, attribute‑labeled reasoning (the so‑called chain‑of‑thought, or CoT, prompts). These CoT traces are then inspected to reconstruct the Q‑matrices that define which attributes each test item requires. The method is designed to help experts check that AI‑generated matrices reflect intended skill sets.
In a simulation study of ten fraction–subtraction items with added noise, four different LLMs—DeepSeek v4 Flash, Claude Haiku 4.5, Kimi K2.6, and GPT‑4o mini—successfully recovered the full target attribute vectors for all items. Three models achieved perfect overlap (1.00) with the intended matrices, while GPT‑4o mini had an overlap of 0.80, missing some attributes because it mislabeled operations.
The same four models were tested on 11 real fraction‑subtraction items answered by 536 examinees. They generated Q‑matrices under three temperature settings, with 100 replications each. Q‑matrices produced by direct generation (without CoT) overlapped the reference matrix on average at 0.877, whereas CoT‑derived matrices overlapped at 0.809. The direct approach was also more stable: the dominant‑matrix proportion—a measure of consistency across replications—was higher than for the CoT workflow.
When inspecting the CoT traces, researchers found that differences in operation order did not affect the final attribute set but caused trace‑analysis variability. Mislabeling of combined steps—e.g., merging a borrowing operation with a whole‑number separation—led GPT‑4o mini to omit key attributes on some perturbed items. These trace‑level variations also showed that different temperature settings and model families produce diverse Q‑matrices, reflecting sensitivity to stochastic sampling and design choices.
Overall, the study shows that LLMs can approximate expert‑specified Q‑matrices, but evaluating the reasoning traces is essential to identify when and why they fail. Future work will refine attribute labeling rules and explore how to combine RTA with tighter expert review to ensure reliable measurement tools.