A practicing surgeon and AI healthcare executive described how incorrect medical information from ChatGPT misled a patient, even as healthcare professionals express cautious support for OpenAI’s plan to introduce a dedicated health-focused chatbot.
Faulty Medical Advice Reaches Patients
Dr. Sina Bari, a practicing surgeon and AI healthcare leader at iMerit, said he recently encountered a patient who challenged a prescribed medication using information generated by ChatGPT. The patient brought a printed conversation stating the medication carried a 45 percent chance of pulmonary embolism.
Dr. Bari told TechCrunch that further review showed the statistic came from a research paper focused on a narrow subgroup of patients with tuberculosis. The data did not apply to the patient’s condition. Despite the incident, Dr. Bari said the experience did not diminish his interest in upcoming changes to the chatbot.
OpenAI Introduces A Dedicated Health Chatbot
Last week, OpenAI announced ChatGPT Health, a version of ChatGPT designed for health-related conversations. The company said the product will roll out in the coming weeks and allow users to discuss health topics in a private setting, where conversations are not used to train underlying AI models.
Dr. Bari said formalizing health conversations with additional privacy protections and safeguards could make the tool more practical for patients, noting that people are already using AI systems for medical questions.
Data Privacy And Regulatory Concerns
ChatGPT Health allows users to upload medical records and connect data from applications such as Apple Health and MyFitnessPal. This capability has raised concerns among data security professionals.
Itai Schwartz, co-founder of data loss prevention firm MIND, told TechCrunch that the movement of medical data from HIPAA-compliant organizations to vendors not covered by HIPAA introduces regulatory uncertainty. Schwartz said it remains unclear how regulators will respond to this type of data transfer.
Widespread Use Of AI For Health Questions
Some industry professionals say widespread use of AI chatbots for health information has already become common practice. More than 230 million people discuss health-related topics with ChatGPT each week, according to figures cited in the report.
Andrew Brackin, a partner at Gradient, told TechCrunch that healthcare questions represent one of the largest use cases for ChatGPT. He said a more private and optimized version of the chatbot aligns with existing user behavior.
Hallucinations And Model Accuracy
AI chatbots continue to face challenges with hallucinations, a concern that carries heightened consequences in healthcare. According to Vectara and its Factual Consistency Evaluation Model, OpenAI’s GPT-5 shows a higher tendency to hallucinate than several models developed by Google and Anthropic.
At the same time, AI companies continue to introduce healthcare-focused tools. Anthropic announced a health-related product this week, adding to a growing list of AI-driven healthcare initiatives.
Provider-Side AI Seen As More Urgent
Dr. Nigam Shah, a professor of medicine at Stanford University and chief data scientist at Stanford Health Care, said limited access to care poses a more immediate issue than incorrect chatbot advice.
Dr. Shah said wait times to see a primary care doctor in the United States can range from three to six months. He framed the choice patients face as waiting months for a physician or turning to an AI system that can provide limited assistance.
Dr. Shah said he sees a clearer path for AI adoption on the provider side. Medical journals have reported that administrative tasks can take up roughly half of a primary care physician’s time, reducing patient capacity.
AI Tools Inside Clinical Systems
Dr. Shah leads a Stanford team developing ChatEHR, a system integrated into electronic health records that allows clinicians to interact with patient data more efficiently.
Dr. Sneha Jain, an early tester of ChatEHR, said in a Stanford Medicine article that simplifying access to medical records could reduce time spent searching through patient files and allow physicians to focus more on patient interaction and diagnosis.
Insurers And Administrative Automation
Anthropic is also developing AI tools intended for clinicians and insurers rather than public-facing chatbots. This week, the company announced Claude for Healthcare, which is designed to reduce time spent on administrative processes such as prior authorization requests.
Anthropic chief product officer Mike Krieger said during a presentation at J.P. Morgan Healthcare Conference that reducing 20 to 30 minutes per authorization case could result in significant time savings for healthcare staff who handle large volumes of requests.
Ongoing Tension Between Medicine And Technology
As AI systems become more integrated into healthcare, Dr. Bari pointed to a structural tension between medical practice and technology companies. He said physicians are primarily accountable to patient safety, while technology firms ultimately answer to shareholders.
Dr. Bari said that this tension reinforces the need for doctors to approach new tools with caution in order to protect patients.
Featured image credits: Freepik
For more stories like it, click the +Follow button at the top of this page to follow us.