Large language models (LLMs), such as ChatGPT and Claude, are increasingly being used to provide health information to the public. Although this is beneficial for improving access to treatment information, their limitations have been revealed by a recent study that found LLMs were biased in their treatment choices and using these outputs in practice could drive up treatment costs.
A research team led by Daiju Ueda at Osaka Metropolitan University’s Graduate School of Medicine in collaboration with The University of Texas’ MD Anderson Cancer Center asked five LLMs (GPT-5.4, Claude Sonnet 4.6, Gemini 3.1 Flash, Llama 4, and MedGemma) to recommend up to three FDA-approved drug products for a range of simulated scenarios.
Although an average of 5.5 FDA-approved products were available for each specified active ingredient, a single product accounted for an average of 97.5% of first-choice recommendations across the tested scenarios and models.
“The recommendations were mostly for brand-name drugs, those highly visible in Google searches, and treatments with dedicated Wikipedia pages,” Ueda said.
The LLMs also tended not to recommend products with later FDA approval and those more commonly prescribed in actual physician practice. The study also identified recommendations that were inconsistent with regulatory information, including discontinued products.
The researchers then evaluated how much this could potentially cost patients using US data. They looked at a scenario where physicians’ prescribing behaviour was performed in accordance with the LLMs’ recommendations.
“We found that annual acquisition costs would increase by approximately $55.6m,” Ueda said.
“This represents a 1,019% increase in the cost of purchasing these prescriptions.”
When instructed to prioritise lower-cost or generic products, the models shifted their recommendations accordingly; however, recommendations remained concentrated on a limited set of products.
“Our findings highlight the need to evaluate healthcare AI for biases in product selection and the handling of cost and regulatory information,” Ueda said.
“Combining AI systems with trusted drug information sources and human oversight may help support more appropriate medication choices.”