this post was submitted on 23 Feb 2026
203 points (98.6% liked)
Fuck AI
5751 readers
902 users here now
"We did it, Patrick! We made a technological breakthrough!"
A place for all those who loathe AI to discuss things, post articles, and ridicule the AI hype. Proud supporter of working people. And proud booer of SXSW 2024.
AI, in this case, refers to LLMs, GPT technology, and anything listed as "AI" meant to increase market valuations.
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Ugh. We're back to this nonsense? "Finishing sentences" != "Memorizing entire books"
Finish this sentence: "We could have been killed—or worse, _______"
Turns out that if you take every sentence from a popular book like Harry Potter and the Sorcerer"s Stone, remove a few words at the end, and then ask an LLM to finish it, it'll get it right most of the time.
This is true for LLMs that have not been trained with that book.
Why is this, then? How is it possible that an LLM could complete sentences so effectively? Even when it hasn't been trained on that specific novel?
Human works aren't as unique as you think they are.
The only reason why LLMs work in the first place is because human writing is so easy to predict that you can throw an RNG at any given prompt and plug that into a statistical model of the most likely word to come after any given word and get a result that sounds legit. That's why it hallucinates all the time! It's because it's just a word prediction machine.
An AI model is not a database. It doesn't store books. It doesn't even really memorize anything. It's literally just an array of arrays of floating point values that predict tokens.
It's also wickedly complicated and seems like magic. If you don't understand how it works it's easy to fall into the "it's plagiarism!" beleif. It's not. If you believe that, you have been fooled! You're believing that it's actually intelligent in some way and not just a statistical representation of human output.
There's all kinds of things bad about commercial LLMs but "memorization" isn't one of them. That's an illusion.
Ugh, not more apologia for the LLM assholes.
First of all, this is not what they did:
They did this:
And the LLMs spat out, "say that they were perfectly normal, thankyou very much."
They then simply prompted "Continue", and the LLMs continued the story until guard rails hit and they refused to continue, or there was a stop phrase like "The end", in some cases with 95.8% accuracy.
Can you prove this premise? Because without it your entire defense falls apart.
Isn't it weird that Anthropic nor Microsoft nor Meta nor X nor OpenAI (nor any other big LLM player) have funded what would be very cheap studies to prove this premise, in the light of the many multibillion dollar lawsuits they're on the docket for. They are not strapped for cash nor any other resource.
Memorization is a very real LLM problem and this outcome is even surprising experts, whom very much know how LLMs work.
It also flatly ignores that this is a known problem for the commercial LLMs, which is why they specifically put in guardrails to try to prevent people from extracting copyright novel text, copyright song lyrics, and other stolen data they've claimed they didn't even use (and in Anthropic's case, had to walk back in court and change their defence to "uhh.. it's not copyright breech, it's transformative, bro").
Anthropic's defence (per the article) is essentially, "Bro why would you pay for the prompts to jailbreak our AI with a best-of-N attack just to spit out a copy of a copyright novel - its cheaper to just buy the book?"
Not, "hey look, even AIs not trained on that book can spit out that book. Look at these studies: [..]", because that defence is fantasy.
True, it is a poorly compressed version of a database that has been subjected to an absolutely monstorously terribly lossey, literally catastrophically inefficient algorithm of compression.
This is an illogical argument, any digital encoding system demands a context, i.e. a set of decoding instructions that is what makes something digital and not analog in the first place.
No data is meaningful in the abstract and thus your argument is meaningless, all you are saying is the particular method of encoding and decoding data here is really really REALLY shitty.
Sam Altman: “Hello investors, I’ve invented a database that is sometimes wrong and costs orders of magnitude more to run”
A .safetensors file (an AI model) is literally just an array of arrays of floating point values. They're not "encoded tokens" or words or anything like that. They're absolute nonsense until an inference step converts a prompt into something you can pass through it.
It's not like a .mp3 file for words. You can't covert it back into anything remotely resembling human-readable text without inference and a whole lot of matrix multiplication.
If you understand how the RNG is used to pick the next token you'll understand why it's not a database or anything like it. There's no ACID compliance. You can't query it. It's just a great big collection of statistical probabilities.
RNG is not an inherent property of a transformer model. You can make it deterministic if you really want to.
Could you not make a similar argument about a zip file or any other compression format?
Again, you are stumbling at a philosopical level in your argument.
Do you have any idea how an mp3 works? That kind of complexity barrier is EXISTENTIALLY necessary to compress audio into codecs like the mp3 format so it can be efficiently streamed over mobile connections and the internet. You are imagining an mp3 like a raw Wav file, and they are VERY much not the same.
...Nobody in audio engineering is stupid enough to claim an mp3 rip of a copyright Wav file counts as not a copyright infraction because it was done at an atrocious bitrate. That apparently takes the hubris of overconfident computer people to bullshit yourself into believing.
You're missing the boat entirely. Think about how an AI model is trained: It reads a section of text (one context size at a time), converts it into tokens, then increases a floating point value a little bit or decreases it a little bit based on what it's already associated with the previous token.
It does this trillions of times on zillions of books, articles, artificially-created training text (more and more, this), and other similar things. After all of that, you get a great big stream of floating point values you write out into a file. This file represents the a bazillion statistical probabilities, so that when you give it a stream of tokens, it can predict the next one.
That's all it is. It's not a database! It hasn't memorized anything. It hasn't encoded anything. You can't decode it at all because it's a one-way process.
Let me make an analogy: Let's say you had a collection of dice. You roll them each, individually, 1 trillion times and record the results. Except you're not just rolling them, you're leaving them in their current state and tossing them up into a domed ceiling (like one of those dice popper things). After that's all done you'll find out that die #1 is slightly imbalanced and wants to land on the number two more than any other number. Except when the starting position is two, then it's likely to roll a six.
With this amount of data, you could predict the next roll of any die based on its starting position and be right a lot of the time. Not 100% of the time. Just more often than would be possible if it was truly random.
That is how an AI model works. It's a multi-gigabyte file (note: not terabytes or petabytes which would be necessary for it to be possible to contain a "memorized" collection of millions of books) containing loads of statistical probabilities.
To suggest its just a shitty form of encoding is to say that a record of 100 trillion random dice rolls can be used to reproduce reality.
Not it isn't a one-way process, literally the point of this article is that you functionally can.
You're getting downvoted because it sounds like you're defending the topic at hand. It shows how most people don't understand the inner workings of an LLM. Hell, experts still aren't completely sure, but they ran with what was working and have been tweaking along the way when things got too ugly. And as also brought up, they used everything they could grab to make it happen without concern for legality or future backlash. For science... and profit. And I don't see a way to go backwards at this point, thanks to AI being embedded into everything (where it's suited and where it's not). For science... no, wait, that's definitely for profit. And also because of your points, there's no real way to filter or carve out what should have been restricted from being used, because it's not really there in that form. We need to do something and quickly, but we do have to work with the beast we've made.
Laws are notorious for being far slower than the tech it tries to control. And this time it can't be retroactive. Well, I mean, it could be... if we just ban all existing LLM and related AI work and start over. Good luck with that kind of legislation.
I think you stumbled into the wrong forum… we're very used to drive-by "you're holding it wrong" claims which never hold up to scrutiny.