First-hand information for everyone
Researchers built an AI assistant that scans draft legislation for corruption risks. The system follows a layered pipeline that parses text, retrieves relevant examples, uses a language model to evaluate provisions, and applies rule‑based mapping of compliance factors.
Across all flags, the recall was 0.43 for Gemma and 0.46 for Sonnet.
The tool did not match any of NAZK’s 32 legal‑collision counts. Parsing problems caused by OCR errors on scanned bills added to the miss rate.
The benchmark measures recall only; precision remains unknown because NAZK reports only selected defects and does not publish absent findings. The recall figures show the system’s ability to reproduce published agency findings rather than its overall accuracy.
The study was conducted using the publicly released draft–conclusion pairs from NAZK and relied on the TRACE framework for architecture design.