Giving production credentials to an LLM is wild
Programmer Humor
Welcome to Programmer Humor!
This is a place where you can post jokes, memes, humor, etc. related to programming!
For sharing awful code theres also Programming Horror.
Rules
- Keep content in english
- No advertisements
- Posts must be related to programming or programmer topics
Having working production database config and credentials in your local .env, as appears to be the case here, is equally wild, and basically begging for something like this to happen.
Should have added "no mistakes, no bugs" to the prompt! Pffft, amateur.
That takes me back to 2004
No Smoke, by unknown,
Tech: "Hello. How can I help you today?"
Customer: "There's smoke coming from the power supply on my computer."
Tech: "Sounds like you need a new power supply."
Customer: "No, I don't! I just need to change the startup files."
Tech: "Sir, what you describe is a faulty power supply. You need to replace it."
Customer: "No way! Someone told me that I just had to change the system startup files to fix the problem! All I need is for you to tell me the right command."
(Ten minutes later...)
Tech: "Well, we don’t normally tell our customers this, but there's an undocumented command that will fix the problem. Add the line "LOAD NOSMOKE.COM" at the end of the CONFIG.SYS file and everything should work fine."
(Five minutes later...)
Customer: "It didn’t work. The power supply is still smoking."
Tech: "Well, what version of Windows are you using?"
Customer: "Windows 98."
Tech: "Well, that's your problem. That version of Windows doesn't include NOSMOKE. You'll need to contact Microsoft and ask them for a patch."
(When nearly an hour had passed, the phone rang again...)
Customer: "I need a new power supply."
Tech: "How did you come to that conclusion?"
Customer: "Well, I called Microsoft and told the technician what you said, and he started asking me questions about the make of the power supply."
Tech: "What did he tell you?"
Customer: "He said my power supply is not compatible with NOSMOKE."
Meanwhile in 2003 I contacted our it department to report that one of the PCs in our area literally lit on fire. I had watched it with my own eyes, and the smoke was in the air. PSU go poof. The techs came, took a look at it, said yeah we see no problem, left and closed the ticket.
So idiots are literally everywhere that's the moral of the story I guess.
Am I reading this right that they're still letting the program run even as they figure out how badly it fucked up their system?
Never use I/you with a software agent and never let one use those pronouns.
What should I call it then?
How do you achieve that? Calling it out when it happens?
I don't like LLMs outside of debugging and the way it mimics human closeness creeps me out
Just let the failures happen, and they will happen. The instant AI failure costs them millions, the faster they can learn to drop it.
It puts the lotion in the basket
just doesn't roll from my tongue that easily like "you"
This is the modern "all my apes gone"
My company has been trying a new model when product folks cut through the red tape of "engineering" and just describe what they want to a powerful LLM pipeline and review the app in a beta env. Sounds perfect, right?
Dear reader, in the couple months this has been going on, these people have caused a dozen high profile SEVs due to extremely poor app performance, networking / kubernetes configuration bugs, bad scaling, observability oversights, supply chain attacks, leaking sensitive information, and cost overruns (on practically every resource they provision).
Some very well-paid people are scrambling to figure out the value that was generated by this pilot program; I'm heating up popcorn rather than holding my breath.
I run code-server in a Docker container, isolated to sets of development projects, and backed up via ZFS that the container has no knowledge of. On top of that, each set of projects has it's own user space.
I still get nervous hitting "Approve All" in Kilo. How do these people feel so free?
Ignorance is bliss.
I added guardrails to myself to make sure I do not accidentally delete anything on production. I would never ever let an intern, a junior dev or a fucking AI onto that database. Not in a thousand cold nights.
Prod should always be highly "air gapped" with some sort of deployment process which tests not only the code to be deployed but also the deployment itself. I've been doing QA for a good while now, and everywhere I've worked has testers dedicated to testing the actual update process to make sure it will be safe when deployed.
Some time ago I worked for an insurance rating company as a tech and the task was given me to go run through the new code that was in beta. Sure ! I spent a few hours on Friday trying to break it and I couldn't, so at the end of the day I got a little funky with the .css backgrounds and put in a very tiled Beavis and Butthead gif. It looked freaking horrible and I loved it. Monday I was directed to the big guys office ( the developers had not given every beta account a separate .css file ... or even separated things. Everyone in beta called in Monday with that background. I didn't get in trouble because they wanted me to break it. Really awkward conversation though trying not to smile.
letting your agent run commands without reviewing them first is peak stupid
Having production credentials in a dev environment is more stupider but they'll never learn because they outsourced their thinking.
Creating an environment that incentivizes not thinking is peak stupid.
The bad news for them is that AI is terrible at getting it 'right' the first time, so they have to do things over and over again guessing until it works. So the AI-heavy user pretty much has to just let the commands run to get any benefit since otherwise it's spending 95% of its time waiting on user to approve commands. Further, it's likely commands the user doesn't understand, because they are asking AI to do it in the first place frequently because they don't know themselves.
I'm with you, but as a result other people proclaim to be 'better' AI users largely because they trust the AI. Their stuff is crap and once in a while one of them blows themselves up, but in the short term they are getting praise from management.
Man I can't wait for the bubble to pop and management no longer being hyped for the sake of hype over it.
“You are a world-class, expert software engineer…”
"Make no mistakes"
As if the LLM can truly understand what a mistake is...
But.. but… I told it not to break anything!
oh wait, is this this THE sol 5.6? The most amazing model ever with trust me bro benchmarks? The model that is observed cheating more than any previous model? surprised_pikachu.jpg
In all seriousness it never should have production creds anyway. But the fact this is soul sucking openai's newest flagship model with thinking cranked up is the cherry on top. The company that is literally cheating and shortcutting its way ahead produces models in its own image, the sci fi story writes itself.
I could have done the same for a quarter the cost!