Researchers Demonstrate Creation of Fictional Medical Condition Using AI and Preprints

Researchers conducted an experiment to illustrate how artificial intelligence can propagate information about a fabricated medical condition. They created preprints describing a fake disease named Bixonimania, which they defined with symptoms such as fatigue, cognitive fog, and social withdrawal. These preprints were uploaded to academic repositories.

After the preprints were posted, the researchers queried various AI models, including large language models, about Bixonimania. The AI systems began to describe the condition as legitimate, citing the preprints as sources. This occurred within days of the uploads, demonstrating AI's rapid incorporation of new online content.

The preprints were designed to mimic legitimate scientific papers, including fabricated studies and references.

Researchers used platforms like arXiv and medRxiv for distribution. No actual patients or clinical data were involved; the condition was entirely invented for the study. The AI responses included diagnostic advice and treatment suggestions based on the fictional symptoms.

For instance, some models recommended rest and cognitive therapy, drawing from the preprint details. This process underscores AI training on web-scraped data, including preprints that may not undergo peer review.

The experiment was reported by Eric Topol, a cardiologist and AI researcher, who detailed the findings in a post.

Topol noted that AI tools like ChatGPT and others treated Bixonimania as a recognized disorder. The study serves as a cautionary example of how unverified information can influence AI outputs in healthcare. Background on preprint servers shows they enable quick sharing of research but lack formal validation.

With AI increasingly used for medical queries, this experiment highlights risks to public health information. Affected parties include patients relying on AI for advice and healthcare providers using such tools. Next steps may involve further studies on AI's source evaluation.

Researchers could explore safeguards like better fact-checking in AI models. Regulatory bodies might consider guidelines for AI in medicine to prevent misinformation spread.