First-hand information for everyone
Researchers introduce “circuits” that organize LLM classifiers into an adaptive sequence of queries and decisions. The goal is to maximize average classification accuracy for a given average query cost. (F1, F2)
A circuit is defined as a pair (π_query, π_predict). The query policy adaptively selects which instruments — LLM classifiers treated as noisy measurements of the true label — to call and when to stop. The decision rule outputs the final label from the history of instrument responses. (F4, F2)
For a single prompt, the optimal circuit policy is the solution to a dynamic program derived from the Bellman equations. The optimal solution can be computed exactly by backward induction over stages. (F3)
The practical implementation, called LMCircuit, uses a two-step approach. First, it fits a single probabilistic model of the joint distribution of the label and instrument outputs from labeled data, using prompt embeddings. Second, it plugs those estimates into the dynamic program to generate a prompt-dependent policy that decides which LLM to query and when to stop. The method assumes that, conditional on a prompt, instrument measurements are independent. (F5, F6)
Experiments evaluate circuits on seven classification benchmarks spanning safety, health, politics, reasoning, human preference and agent evaluation (AgentJudgeBench, BeaverTails, BIG‑Bench Hard, Decision Index, HealthBench, LMArena and PolitiCause). (F7)
Across these seven tasks, learned circuits (LMCircuit) push out the cost–accuracy frontier relative to routing and cascade baselines. They attain higher accuracy than each individual instrument and than routing/cascade baselines at nearly all budget levels. Benchmarks included predictive routers (RouteLLM, HybridLLM, CARROT), several cascade methods, sampling methods, individual instruments, and majority voting. (F8, F9)
Concrete performance and calibration results: - LMCircuit is fast to train. Across the seven tasks it trains in about 1–3 minutes, while some baselines require minutes to hours. (F10) - The fitted probabilistic model predicts instrument behavior well on unseen prompts. Implied correlations between instruments’ answers and errors are within 4%–17% of measured values. Implied accuracies are within 1.2%–3.6% of measured accuracies across tasks. (F11) - On unseen prompts that are “noisy” (recorded calls correct on a fraction in (1/2,1)), a circuit constrained to a budget of 1 call improves embedding-only prediction by 5%–44%. On prompts where every recorded call is correct, the circuit matches majority voting within 1.3% accuracy while using 38%–86% fewer calls. (F12) - LMCircuit can output a zero‑cost prior prediction using only the prompt embedding (Pr(Y=1|z) ≈ η_θ(ϕ(z),s_0)). This lets the fitted model run entirely on CPU and provide lightweight predictions without querying instruments. (F13)
Fundamental limits and value of instruments: - A zero-cost circuit (no queries) is optimal for a prompt z exactly when the cost parameter λ exceeds a threshold λ_LB(z); a simple sufficient condition is λ ≥ 1/(2 c_min(z)). (F17) - The best possible error using all instruments is the Bayes error after observing the full instrument pool. That minimum error is a lower bound on what any circuit built from that pool can achieve. (F18) - The expected gain from adding a new instrument is bounded by a mutual-information term. Thus an instrument’s value depends on the extra information it provides beyond existing measurements, not merely its standalone accuracy. (F19)
Caveats and limitations: - The dynamic program grows exponentially with the number of models. Large pools can thus create computational scaling limits. Pool construction must trade off added information against the statistical and computational cost of estimating joint behavior. (F14) - Some asymptotic results rely on assumptions that each instrument can be queried arbitrarily often, each instrument is at least as accurate as random guessing on every prompt (accuracy in [1/2,1)), and the response distribution is symmetric conditional on the ground truth. Under these assumptions, per‑prompt optimal error decays exponentially with expected cost at a rate determined by the maximal information rate I(z). (F15) - Averaged across prompts, error–cost scaling depends on the distribution of I(z). Depending on tail behavior, population-level decay can be exponential, stretched-exponential, or polynomial. Exponential per‑prompt decay need not imply exponential decay on average. (F16)
In short, circuits frame LLM classifiers as noisy instruments and provide an adaptive, computable way to trade off cost and accuracy. The approach is practical, fast to fit, and improves cost–accuracy tradeoffs on multiple benchmarks, subject to the stated modeling assumptions and computational scaling limits. (F1, F3, F5, F8)