First-hand information for everyone
What happens when large language models (LLMs) face economic choices? This paper tests whether LLMs behave like humans in decisions about preferences and beliefs. The authors find a clear pattern: on preference questions, more advanced or larger models give answers that look more human-like and often violate classical rational rules. On belief questions, the most advanced large models tend to give more rational, statistically correct answers.
The team adapted experimental questions from cognitive psychology and experimental economics. These are the same kinds of tests researchers use to document human biases. They sent those prompts through application programming interfaces (APIs) to four LLM families: OpenAI’s ChatGPT (including GPT-4 and variants), Anthropic’s Claude, Google’s Gemini, and Meta’s Llama. For each family they compared older and newer model versions, and for the newer versions they compared large-scale and smaller-scale variants.
The results are concrete. In preference-based tasks, Claude3Opus (an advanced large-scale Claude model) matched human-style answers on four of six questions. Claude3Haiku (a smaller advanced Claude) matched on three of six, and the older Claude2 matched on one of six. For belief-based tasks, Gemini1.5Pro (a large advanced Gemini) answered all ten belief questions correctly, while Gemini1.5Flash (smaller) answered five and Gemini1.0Pro (older) answered two. The authors also report variation across model families: for example, Gemini’s preference answers were more human-like than ChatGPT’s, and Meta Llama’s belief answers were less rational than GPT’s.
The paper also re-runs two economics experiments. In a forecasting task that follows Afrouzi et al. (2023), advanced small-scale models such as GPT-4o, Claude3Haiku, and Gemini1.5Flash acted like humans and overestimated how persistent a process was. Their larger-scale counterparts produced forecasts closer to the true persistence. In a stock-investment task following Bose et al. (2022), large-scale models (GPT-4, Claude3Opus, Gemini1.5Pro) made investment choices that depended more on the visual salience of price paths, a human-like bias. Note that six of the twelve models tested could not process graphical inputs, so those tasks were not run on every model.
The authors test ways to reduce biases. A simple “role-priming” prompt that asks the model to think of itself as a rational investor following the Expected Utility framework made model answers more rational and less human-like. The improvement was modest, and other correction methods that added more information did not help. The paper offers two plausible reasons for the patterns: reinforcement learning from human feedback (RLHF) may push advanced models toward human-like preferences, while larger training data and compute may help models find statistical truths for belief questions.
Important caveats apply. The study is descriptive and does not fully identify the mechanisms causing these patterns. The work uses models that existed when the study began in 2023, so results may change as models evolve. Some experiments could not run on all models because of input limitations. Finally, the debiasing gains from role priming were modest, so correcting these behavioral patterns remains a hard problem. The authors make their work a starting point for a public database to track LLM behavioral biases over time.