Can ChatGPT answer health questions better than your doctor?

PHOENIX (3TV/CBS 5) - You have a quick question for your doctor but instead of waiting for an appointment, you send a message through your online patient portal. “Patients are sending more messages than ever before, and increasing messages means more time spent doing work after work hours,” said Dr. John Ayers, a digital epidemiologist. He wanted to know if artificial intelligence could handle some of those questions, so he put it to the test, pitting doctors against ChatGPT.

In a blind comparison using 195 random patient questions from social media, medical professionals compared doctors’ answers to responses generated by a chatbot. “ChatGPT won in a landslide,” Dr. Ayers said. “ChatGPT was nearly four times more likely to write a response that was good or very good quality, and Chat GPT was 10 times more likely to write a response that was empathetic or very empathetic.”

In one example, a person asked about the potential for blindness after accidentally splashing bleach in their eye. On the social media forum, a physician responded “Sounds like you will be fine. You should flush the eye anytime you get a chemical or foreign body in the eye.” The physician also posted the phone number for poison control. ChatGPT’s response was longer and began with a more empathetic tone.

“I’m sorry to hear that you got bleach splashed in your eye,” the chatbot wrote. “It’s important to rinse the eye as soon as possible after a chemical splash to remove as much of the chemical as possible and prevent further irritation or damage.” The AI bot went on to include additional guidance about what to do if the person does not have access to water. Researchers say they weren’t expecting the results.

“We were certainly surprised. Our team, especially our evaluators, were like, ‘There’s no way. We’re going to kick the butt out of AI,’” Dr. Ayers said. “What’s so shocking is the massive developments of this technology. Using a public AI--one that wasn’t tailored to deliver healthcare and wasn’t told to focus on healthcare data when determining its responses--it still beat physicians.”

Dr. Craig Norquist, chief medical information officer at HonorHealth, is skeptical of the study. “At some point, we may be using these models, but we in health care need to use extreme caution,” Dr. Norquist said. “As a trained healthcare provider, we’re trained to ask certain questions to tease out if is this a concerning headache or is this a regular headache. Can you take some Tylenol, take a nap, cut down on caffeine, do any number of things, or do you need to get to the emergency department right away and get your head scanned because you could potentially have a ruptured aneurysm, which would be incredibly tragic if someone relied on a tool that didn’t have all the information.”

Researchers are encouraging more studies. “We believe this is a game changer for health care,” Dr. Ayers said. “Instead of doctors spending time worrying about noun verb conjugation, they can start with a drafted message from ChatGPT or another AI system and then refine that.” The potential applications of large language models extend beyond the medical field. Another recent study showed ChatGPT picked stocks and outperformed several popular investment funds in the UK. Still, the technology is still in its infancy and there are concerns about misleading and false information.

“The word we use is hallucinating,” said Subodha Kumar, director of the Center for Business Analytics and Disruptive Technologies at Temple University. “It will take data from different sources, but based on the information, it can also generate information, and because of that it has generated references or web links that don’t even exist. That makes it very strong and very powerful, but it also makes it very dangerous.” The White House vowed to release guidance in the coming months about how federal agencies will be able to use AI.

Copyright 2023 KTVK/KPHO. All rights reserved.