Fuck AI

Are you saying LLMs can NEVER be safe, because you can always bury their safety rails in context?

For a serious answer?

It's not like humans follow safety instructions all the time either. A machine built to imitate humans will be prone to mistakes too. Any safety from within a system are liable to fail if the system itself makes a mistake.
LLMs have additional limitations compared to humans, but even if we developed and attached some technical model of semantics and morality, the resulting models would at best be an approximation.

For tasks where that safety or accuracy doesn't matter as much, that's fine. Pen-testing a system or hunting bugs doesn't need to strictly abide by certain rules (and in fact may profit from being unconstrained by the assumptions of engineers). Scanning a body of text for irregular patterns or phrases correlating with some keywords where you don't know the exact words could be useful in covering a lot more than a human might immediately think off.

It becomes a problem where accuracy does matter. Scientific work, legal matters, anything involving human safety, sensitive data for example. Here, the risk is that they can make a lot of mistakes much faster. Particularly with large-scale tasks, thoroughly checking the results can be tedious, exhausting and tempting to just shortcut and trust the machine (and you know how lazy people tend to get with tedious, exhausting tasks, so it would be irresponsible to assume they'd be on the ball here).

This is where safety instructions can become dangerous to rely on, because the same unreliability that produces mistakes may also discard the safety instructions.
Letting Claude manage your production environment with its "safety rails" only coming in the form of instructions what not to do is a timebomb for getting your production systems messed up.

Effective safeguards have to come from outside whatever they're supposed to constrain. Supervisors that are supposed to monitor their workers' adherence to protocols, for instance, or mechanisms that physically make mistakes harder (like locking out, tag out).

No safeguards are absolute, and as the saying goes, whenever you invent a foolproof system, the universe invents a greater fool. Stuff like putting a zip-tie around dead man switches on tools or machines that are supposed to shut off the machine if the operator loses control for whatever reason, because it's inconvenient having to keep holding the button and much easier to tell yourself "it'll be fine" (until it isn't).

In some cases, it can also be really difficult to handle certain dangers. But it's worth trying at least, and that requires first acknowledging those problems. In the case or LLMs, that means acknowledging their limits and pitfalls, transparently and honestly communicating them and-

the sad result will probably be to absolve AI companies of liability of things they can't control

That... wasn't my plan, but I'm too poor for my plans to matter anyway. Valid point.

the other option would be to legislate chatbots out of existence

Well, maybe with education and transparency, we could...

Who am I kidding.

how do we get to the utopia of a workerless society?

I propose an experiment where everyone working for the people peddling that dream stops doing so. I haven't worked out the details yet, but I think they should get the chance to experience that utopia. ;-)