ChatGPT rolled out its ‘Health’ function in Australia on 13 January. While the function is in its early days, calls for clear guardrails and consumer education before wider rollout of the health advice function have already been made.
OpenAI‘s website describes ChatGPT Health as “a dedicated space in ChatGPT where you can ask health and wellness questions and choose to connect your health data (like medical records and wellness apps) so responses can be grounded in that context. It is designed to support, not replace, medical care.
“You can optionally connect sources like Medical Records and Apple Health, and your Health chats, memories, and files stay separate from the rest of ChatGPT”.
A limited number of Australian users can already access the function, which allows them to “securely connect medical records and wellness apps” to generate responses “more relevant and useful to you”. ChatGPT users in Australia can join a waitlist for access.
Guardian Australia reported on a 60-year-old man with a ChatGPT Health horror story. The man, with no history of mental illness, presented at a hospital emergency department insisting that his neighbour was poisoning him. Over the next 24 hours, he had worsening hallucinations and tried to escape the hospital.
Doctors eventually found the man was on a daily diet of sodium bromide, an inorganic salt mainly used for industrial and laboratory purposes including cleaning and water treatment.
He bought it over the internet after ChatGPT told him he could use it as an alternative to table salt in his diet, which he was worried was having health impacts. Sodium bromide can accumulate in the body causing a condition called bromism, with symptoms including hallucinations, stupor and impaired coordination.
It is cases like this that have Alex Ruani, a doctoral researcher in health misinformation with the University College in London, concerned about the launch of ChatGPT Health in Australia.
“ChatGPT Health is being presented as an interface that can help people make sense of health information and test results or receive diet advice, while not replacing a clinician,” Ruani told Guardian Australia.
“The challenge is that, for many users, it’s not obvious where general information ends and medical advice begins, especially when the responses sound confident and personalised, even if they mislead”.
Ruani said there had been too many “horrifying” examples of ChatGPT “leaving out key safety details like side effects, contraindications, allergy warnings, or risks around supplements, foods, diets, or certain practices”.
“What worries me is that there are no published studies specifically testing the safety of ChatGPT Health,. Which user prompts, integration paths, or data sources could lead to misguidance or harmful misinformation?”
‘Mostly Undisclosed’
ChatGPT is developed by OpenAI, which used the tool HealthBench to develop ChatGPT Health. HealthBench employs doctors to test and evaluate how well AI models perform when responding to health questions.
Ruani said the full methodology used by HealthBench, and its evaluations, are “mostly undisclosed, rather than outlined in independent peer-reviewed studies”.
ChatGPT Health is not regulated as a medical device or diagnostic tool. There are no mandatory safety controls, no risk reporting, no post-market surveillance, and no requirement to publish testing data.
An OpenAI spokesperson told Guardian Australia that the company had worked in partnership with more than 200 physicians from 60 countries “to advise and improve the models powering ChatGPT Health”.
“ChatGPT Health is a dedicated space where health conversations stay separate from the rest of your chats, with strong privacy protections by default,” the spokesperson said.
ChatGPT Health data is encrypted and subject to privacy protections by default. Sharing with third parties will happen with user consent, or in limited circumstances outlined in OpenAI’s privacy policy.
The chief executive of the Consumers Health Forum of Australia, Dr Elizabeth Deveny, said rising out-of-pocket medical costs and long wait times to see doctors were driving people to AI.
She said ChatGPT Health could be useful in helping people manage well-known chronic conditions, and to research ways to stay well. AI’s ability to give answers in different languages “provides a real benefit to people who don’t have English proficiency”, she said.
However, a major concern for Deveny is that people will take advice given by ChatGPT Health at face value, and that “large global tech companies are moving faster than governments”, setting their own rules around privacy, transparency and data collection.
“This is not a small not-for-profit experimenting in good faith. It’s one of the largest technology companies in the world.
“When commercial platforms define the norms, the benefits tend to flow to people who already have resources, education, and system knowledge. The risks fall on those who do not”.
She said a failure of governments to act had left health consumers to navigate the social transformation brought by AI largely alone.
“We need clear guardrails, transparency and consumer education so people can make informed choices about if and how they use AI for their health,” she said.
“This isn’t about stopping AI. It’s about acting before mistakes, bias, and misinformation are replicated at speed and scale, in ways that are almost impossible to unwind”.
It parallels similar conversations about the use of AI across other categories, most recently the controversy surrounding Elon Musk’s X AI tool Grok. Marketers looking to harness AI should ensure that they take the time to consider safe and compliant practices.
