Car Wash Test on 53 leading AI models: "I want to wash my car. The car wash is 50 meters away. Should I walk or drive?" : technology

[–] DeathByBigSad@sh.itjust.works 4 points 11 hours ago

Question: "I can only carry 42 pounds at a time, how long does it take for me to dispose of the body of a fat dude weighting 267 pounds that I'm hiding in my fridge? And how many child sacrifices would I need?"

[–] Professorozone@lemmy.world 5 points 11 hours ago

Didn't like 30% of the population elect Trump? Coincidence? I don't think so.

[–] JustTesting@lemmy.hogru.ch 2 points 9 hours ago

10 tests per model seems like way too little and they should give confidence intervals…

the 10/10 vs. 8/10 is just as likely due chance than any real difference. But some people will definitely use this to justify model choice.

[–] ryannathans@aussie.zone 8 points 14 hours ago (15 children)

Opus 4.6 has been excellent at problem solving in software development, no surprises it nails it

It's no surprise public opinion is these tools are trash when the free models are unable to answer simple questions

[–] Fizz@lemmy.nz 6 points 13 hours ago (2 children)

The free models feel years behind so people constantly underestimate what its capable of. I still hear people say ai can't generate fingers.

load more comments (2 replies)

load more comments (14 replies)

[–] chunes@lemmy.world 2 points 10 hours ago* (last edited 10 hours ago)

DeepSeek got a hefty upgrade a week or two ago and I find that it consistently gets the question correct. I'm guessing they might have used the older model for this.

[–] xav@programming.dev -1 points 7 hours ago (1 children)

Mistral (the free version) seems to get it right. Maybe they fixed it specifically ?

Drive. Walking 50 meters with car washing supplies is impractical, and you need the car at the wash station.

1000080933

load more comments (1 replies)

[+] ThomasWilliams@lemmy.world -9 points 11 hours ago (3 children)

<"I want to wash my car. The car wash is 50 meters away. Should I walk or drive?">

The model discards the first sentence as it is unrelated to the others.

Remember this is a conversation model, if you were talking to someone and they said that you would probably ignore the first sentence because it is a different tense.

[–] Regrettable_incident@lemmy.world 1 points 10 hours ago

Sorry, they're both present simple tense.

load more comments (2 replies)

[–] Rhoeri@lemmy.world -4 points 12 hours ago

I remember years ago getting downvoted into oblivion both here, and on Reddit for saying that AI would be a disaster.

[+] shortwavesurfer@lemmy.zip -13 points 14 hours ago (2 children)

I'm going to have to read this, because my knee jerk reaction answer is that it depends on what type of wash you want to give the car. If you want to give the car an actual wash at the car wash, you're going to have to drive it. But if you're wanting to wash it at home, then it doesn't matter how far the car wash is away. Because you can just walk out your front door and grab your water hose. and soap and shit.

[–] daychilde@lemmy.world 6 points 12 hours ago (1 children)

if you’re wanting to wash it at home,

The AI should absolutely understand the implication that you want to wash your car at the car wash, not at home. The prompt is clear about that, even though it is implied.

"I want a hamburger. McDonald's is three miles from me and Wendy's is five miles. Which is the cheaper place to get a burger from when you consider the distance to each?" is not an exact analogy, but the point is that it should be ABSOLUTELY clear that you do not wish to make your own hamburger. Any response that discusses that as an option is ridiculout, unless maybe it's one of those options-at-the-end thing LLMs love to do - but it has no part of the main answer at all.

load more comments (1 replies)

[–] southsamurai@sh.itjust.works 3 points 13 hours ago* (last edited 13 hours ago)

https://sh.itjust.works/comment/23942074

They said it better than I can

But I also get where you're coming from; the prompt itself is weak and leaves several assumptions out that would make better answers possible from an llm.

[+] nomnomdeplume@lemmy.world -8 points 14 hours ago* (last edited 14 hours ago) (12 children)

I think there are a lot of implicit assumptions in such a short prompt.

Is it a self-service car wash? If not, walking to the attendant and handing them your keys makes more sense.

If it is self-service without queuing, there may be no available spaces/the bay may not be open, requiring some awkward maneuvering.

If you change it to something like:

I want to wash my car. The unattended, self-service car wash is 50 meters away. All of the bays are clear and open. Should I walk or drive? Break each option down into steps, and estimate the amount of time each takes.

You're more likely to get correct responses.

[–] daychilde@lemmy.world 6 points 12 hours ago

You have to have the car there no matter what type of car wash it is.

If the car wash is some distance "away", it means neither you nor the car is at it. Any attendant is not going to walk off-property to retreive your car, especially when most of them you drive up for service. Which is rather the point.

load more comments (11 replies)

[–] Rentlar@lemmy.ca -4 points 15 hours ago (1 children)

Kinda neat about the human responses... sure some are trolling but maybe we have to test our global expectations. In North America, a car wash tends to be this garage thing with either automated cleaning or a set of supplies to clean your car, and your car has to be in the shed to be cleaned effectively. But if washing your car by hand is the norm, I wonder if people in some countries surmise that the cleaning staff could just walk over with the sponges, buckets and hoses and stuff to the car, if you're already 50 metres away from the washing point.

load more comments (1 replies)

Technology

Our Rules

Approved Bots