Technology

Yudkowsky tends to like engineering bioweapons in If Anyone Builds It, Eveeyone Dies.

I'm not sure that that'd be my first concern, but I also don't really think that ability to affect the non-electronic world is really the big hurdle. I think that the bug hurdle is getting some kind of AGI running somewhere, and once that happens, it's going to be really difficult to keep it from acting in the real world in a way that gives itself more capability to act.

Like, I don't think that the idea that computers are a sort of effective containment system and that the AI is inside a computer, and humans will act as a sensible check on the computer influencing the real world, so it can't do much, holds up very well. If you do wind up with something more-capable than humans running around, I think that it's going to be very hard to anticipate all the possible risks.

Say that someone manages to get an AI producing something particularly valuable and useful, doing something profitable for them. My guess is that they are probably going to link it up to whatever they think will be useful.

My understanding from past reading is that one of China's principal near-term interests is in trying to leverage AI to do industrial automation. Even if it were not, you're going to have more-and-more heavily-automated production facilities, and probably more general-purpose systems.

https://en.wikipedia.org/wiki/Lights_out_(manufacturing)

Lights-out manufacturing or dark factory is the manufacturing methodology of fully automating the production of goods at factories and other industrial facilities, without requiring any human labour presence on-site.[1] Many of these factories are considered to be able to run "with the lights off," but few run exclusively lights-out production. For example, in computer numerical control machining, the presence of human workers is typically required for removing completed parts and setting up tombstones that hold unfinished parts. As the technology necessary for total automation becomes increasingly available, many factories are beginning to use lights-out production between shifts (or as a separate shift) to meet increasing production demand or to save money on labor.

An automatic factory is a place where raw materials enter, and finished products leave with little or no human intervention.[2

You have the ability to influence humans, which can also affect the real world. One of the most-seriously disrupted industries by LLMs right now is in copywriting, targeted advertising. Why? Because they're pretty good at influencing humans, even in very primitive form. And we have an interest in building models that can influence people even more effectively.

You don't have to have a human be off trying to intentionally trying to destroy humanity to be doing sonething that gives an AI increasing ability to act in the real world.

I remember reading an article a while back about some guy who had spent a while talking to ChatGPT or some similar chatbot and who became convinced that he needed to violently liberate the god being held captive in the computer. Like, if, as some have put it, "glorified auto-complete" can get a human to do that, what could a much-more-capable system with goals and intentions do? I think that just the ability to communicate with humans, which we certainly already permit (and is one of our main uses), would make containment pretty hard.

AI missile detection system gives false data during heightened conflict and a human decides to push the big red button before getting proper verification. Doesn't have to be AI launching nukes it could be our tools to know when to launch nukes hallucinate and humans make a snap decision