this post was submitted on 25 Jul 2026
1261 points (95.3% liked)
Fuck AI
8190 readers
2535 users here now
"We did it, Patrick! We made a technological breakthrough!"
A place for all those who loathe AI to discuss things, post articles, and ridicule the AI hype. Proud supporter of working people. And proud booer of SXSW 2024.
AI, in this case, refers to LLMs, GPT technology, and anything listed as "AI" meant to increase market valuations.
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Limiting medical knowledge from reliable information resources would be terrible. I’d say that limiting the spread of plausible-sounding medical misinformation generated by a fancy autosuggest tool is good, though.
AI chatbots don’t actually “know” anything. They just generate the most statistically probable appearance of a response to each prompt based on their training data. They don’t tell you when something wasn’t in their training data or when there’s limited research on a subject, just give you the most statistically probable response like always, which leads to a “hallucination” that sounds true but isn’t. A tool that’s trained on MCAT testbanks and “aces” the MCAT won’t necessarily give reliable medical advice to real patient questions. Even if you gave an AI chatbot all the accurate training data in the world, it would still hallucinate information occasionally, and it wouldn’t give you any indication of when it did so. If your doctor screws you over, they can lose their license, but if ChatGPT screws you over, there’s currently very little to stop OpenAI from screwing over more people. OpenAI can just stand behind the disclaimer that says ChatGPT can make mistakes, never mind that users are treating it like a licensed medical professional. There are AI tools that are useful in medical research, but they’re a VERY different kind of AI from the AI chatbots this bill is concerned with.
If you want medical knowledge, there are tons of medical journals, news sources, blogs, and podcasts from doctors/scientists who don’t hallucinate BS when they don’t know something.
Instead of banning it then teach people how to use it and not to trust it 100% (who the hell is doing this to begin with?)
I can ask an LLM to help me sourcing these journals and informations
Expecting people who have already taken the easy route of asking an AI to summarize medical information to do additional due diligence by reading the sources that the AI spits out is naive as fuck.
There's a reason it became a meme for someone to Google their symptoms, click on WebMD to read about it, and then come to the conclusion that they have cancer, all without ever having seen a medical professional in the meanwhile. Do you really think these sorts of people are going to be using AI responsibly for research purposes only?
If they come to the wrong conclusion at worst they go to the hospital and someone there tells they are wrong and not to worry. We haven't ban googling your symptoms.
You've missed the point.
Someone who Googled their symptoms and ended up at WebMD were presented with reliable albeit ambiguous information. Carefully curated by medical professionals and not random snake oil salesmen or homeopathy gurus. They fail to interpret the symptoms correctly, because of course they would. If they are looking up causes of a headache, it can be anything from brain tumor to low blood sugar, or even no reason at all.
Someone who asks AI for medical advice is going to get the entire internet worth of fucking opinions, as well as medical journals and such, because that's where ChatGPT, Gemini, Grok and Claude all got their data from. And it will string those words together in a way that sounds authoritative, but it will have literally zero concept of what it's talking about. It's an overgrown autocomplete. You ask for it to provide some citations, but no amount of asking for citations will compel people to actually read them, nor will there be any guarantee that the LLMs couldn't just hallucinate that as well.
It's like taking a Google search for medical information and adding a middleman who occasionally filters in junk data, then wrapping it in a nice package, giving the impression of authority and accuracy, making the process as frictionless as possible.
There can be no reasonable expectation that people who have not historically been capable of making good choices in regards to seeking online medical advice will suddenly start doing it all of their own volition.
You might say "What's the harm if they end up at the hospital either way"? But the danger is that they never go to the hospital at all because the chatbot told them to sun their perineum or inject bleach with the confident prose of a medical doctor and they just... do it. Because why would the AI be wrong?
And who is to blame when someone is injured or killed as a result? Doctors have standards, and they take an oath to first do no harm. If they make a mistake, they are liable for it. How can you hold a machine intelligence accountable for inaccurate diagnoses?
The simple answer is you can't, and so logically they should be strictly prohibited from masquerading as any kind of authority on medicine.
If LLM answers had zero concept they wouldn't work. We are here talking about it because the answers are correct up to a certain point that makes them good enough.
These middleman are also called teachers, when you go to medical school you have many of these, you don't get handled the medical encyclopedia and left on your own.
that's something you are making up, an LLM telling you to inject bleach would be the same as it telling you that 2+2= coconut, it's the wrong answer, if all the answers where wrong we wouldn't be here talking about this tech.
Doctors prescribe you medicines and do surgery on you, an LLM can't do these things, if AI tells you to take a medicine you still have to go to the medic to get these.
The propaganda meme is talking about a ban on AI answers about medicine. LLMs have no authority on medicine, that's something we will be discussing in 20 years.
You can teach people how to use it all day long, but most still won’t use it well consistently in real life. Humans instinctively trust confident-sounding language. Most of us are also way worse at fact checking than we think. LLMs play right into humanity’s flaws, and they’re built to maximize engagement rather than help us avoid going past the tools’ limits. I don’t trust LLMs to be accurate about anything, but I still caught myself reading and failing to check Google AI overviews sometimes before I changed my default search engine to a no-AI alternative. It’s just instinctual when an “answer” is right there at the top of the page in response to your search question.
Sure, you can use an LLM to help you find sources, but at that point you might as well just use a search engine. There are search engines that provide links with actual text from each site rather than potentially hallucinated summaries.
Some popular search engines results are way worst than AI
Some search engine results are crap, yes. But even the worse search engine (minus AI overviews or whatever) gives you a setup where: (1) each piece of information has an actual source, (2) you know who the source is, and (3) you can make decisions about which information to trust accordingly. If a popular search engine puts a sketchy company’s blog at the top of the SERP, you can actually see that and choose to look father down for something more reliable. Also, if your current search engine keeps failing you, try a different one! There are many options out there that you might find more helpful than today’s Google or Bing or DuckDuckGo.
Good luck with things, truly. I think you’re going to get hurt eventually if you use LLMs for medical advice, and I genuinely don’t want that for you or anyone. Hopefully this conversation has at least made you a little more skeptical about using LLMs in certain ways.
Good luck with things too, in case you haven't noticed this meme is from a weird association that doesn't seem to exist and that promotes pseudo scientific content
Great! Let's add regulation/accountability to search engine results too then, if they're as problematic as this.
I don't know where you live but in my country people don't go to the medic or hospital for minor things because it cost them money and time. In this scenario limiting access of any sort of medical tool and knowledge sounds like further fueling a corrupted system where power is concentrating in the hands of a few individuals and where people are losing their rights.
A 20% bad medical advice is worse than 100% bad advice or no advice.
There's no "teach people how to use it". You either use it and get completely won't advice 20% of the time or you use reputable sources.
To verify what LLM says takes same or more time than just using reputable sources.