this post was submitted on 07 Mar 2026
845 points (99.1% liked)
Technology
82494 readers
4564 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Here's the source it's from open AI but it is peer reviewed. Here's another source that uses it as a baseline to compare the relative scores and according to the tables in 2023 it got a 610, putting it around the 75th percentile, and that's just for math which the open AI study showed it did about 5% worse then it's average so ~80th percentile for a total score. Again this is for students who are usually more prepared for the SAT than the general population, so it's still probably in the 90th percentile for the general population.
Again for the car wash example that is not declaritive knowledge, like the pizza glue that is knowledge derived from experience and reason which I've said that LLMs aren't the best at. The fact that they had to make a riddle for the AI to trip it up if anything shows how good it is. If it was as bad as you say it is then anyone could easily trip it up and get it to give a wrong answer and a study like that wouldn't be relevant. Seriously if you think the LLM is so inaccurate, come up with your own test to stump it, it should be easy by the way you talk about them.
"I want to take my car to the car wash, should I walk or drive" is not a riddle. It requests basic understanding of what is being asked.