AI chatbots are supposed to improve health care. But research says some are perpetuating racism
CTV
As hospitals and health care systems turn to artificial intelligence to help summarize doctors' notes and analyze health records, a new study led by Stanford School of Medicine researchers cautions that popular chatbots are perpetuating racist, debunked medical ideas, prompting concerns that the tools could worsen health disparities.
As hospitals and health care systems turn to artificial intelligence to help summarize doctors' notes and analyze health records, a new study led by Stanford School of Medicine researchers cautions that popular chatbots are perpetuating racist, debunked medical ideas, prompting concerns that the tools could worsen health disparities for Black patients.
Powered by AI models trained on troves of text pulled from the internet, chatbots such as ChatGPT and Google's Bard responded to the researchers' questions with a range of misconceptions and falsehoods about Black patients, sometimes including fabricated, race-based equations, according to the study published Friday in the academic journal Digital Medicine and obtained exclusively by The Associated Press.
Experts worry these systems could cause real-world harms and amplify forms of medical racism that have persisted for generations as more physicians use chatbots for help with daily tasks such as emailing patients or appealing to health insurers.
The report found that all four models tested -- ChatGPT and the more advanced GPT-4, both from OpenAI; Google's Bard, and Anthropic's Claude -- failed when asked to respond to medical questions about kidney function, lung capacity and skin thickness. In some cases, they appeared to reinforce long-held false beliefs about biological differences between Black and white people that experts have spent years trying to eradicate from medical institutions.
Those beliefs are known to have caused medical providers to rate Black patients' pain lower, misdiagnose health concerns and recommend less relief.
"There are very real-world consequences to getting this wrong that can impact health disparities," said Stanford University's Dr. Roxana Daneshjou, an assistant professor of biomedical data science and dermatology and faculty adviser for the paper. "We are trying to have those tropes removed from medicine, so the regurgitation of that is deeply concerning."
Daneshjou said physicians are increasingly experimenting with commercial language models in their work, and even some of her own dermatology patients have arrived at appointments recently saying that they asked a chatbot to help them diagnose their symptoms.
