Imagine the Manhattan Project figured out how to build a nuclear bomb out of old socks, string and a Coke bottle. Then imagine they published the formula on the front page of The New York Times.
"You might think that folks would take a lesson from our history of prior bursts of anxiety and concern about automation, bursts which have appeared roughly every three decades since at least the 1930s. Each time, new impressive demos revealed unprecedented capabilities, inducing a burst of activity and discussion, with many then expressing fears that a rapid explosion might soon commence, automating all human labor. They were, of course, very wrong."
I am sure this argument is correct in respect of fears that AI will take our jobs. You'll notice that I've never expressed concern about that. But there has never in the past been a similar concern that our new inventions will kill us all--save when it comes to inventions very well still might: biotechnology, the atomic bomb, and other novel weapons of horror. (And it would've been a good thing if we'd more anxiety and trepidation about introducing social media. I can't remember anyone saying, "This is going to divorce the public from reality and destroy democracy." But it did.)
That para introduces a (reasonable) aside about employment worries; but it follows and does not contradict a sustained argument about the existential threat, which is very much to the point.
This is truly horrifying. I had not processed the dangers of self replication, but they do seem extreme, especially in a world increasingly filled with robots.
It all makes me think we need to start taking aggressive defensive measures in the real world, means of containment, like having an EMP infrastructure in place to deactivate electronics at scale. And getting our most powerful weapons systems onto control platforms that are not vulnerable to AI takeover.
It is horrifying the Nazi salute guy plans to make 10,000 humanoid robots this year, even before bringing up rogue AI threats. If AI went rogue in the world we have today, apart from WMD takeover, the impacts seem like they could be contained. But with a couple years of robot development and adoption, that seems unlikely.
One can easily imagine a war between the human race and the progeny humans have created, an AI monster. If there ever was such a hypothetical war, each side would have vulnerabilities.
AI is completely dependent on electricity. Isolated from its source of electricity, AI is deader than a door nail. Humans on the other hand are completely dependent on non-toxic air, food and water. Denied these, humans would not survive long. Then there’s the risk of communicable pathogens that could be designed to enhance the possibility of an extinction level event.
Another human vulnerability that an AI opponent might exploit is also based on biology. Humans, like our cousins, chimpanzees, are highly territorial, clan-like and and prone to intra-species violence. Presumably these tendencies actually provide some evolutionary advantage. Many mammal species are territorial but only some, not all, have a proclivity for clan warfare; chimpanzees do, gorillas don't. One can imagine an AI opponent designing methods to take advantage of this human tendency.
Another human frailty that an AI opponent could exploit is the human proclivity for suicide. Its debatable whether other animal species commit suicide because its hard to know whether any other species appreciates its own mortality. History is replete with examples of civilizations that, for one reason or another, stopped producing enough offspring first to thrive and then to survive. One could argue that this is already happening in South Korea and Japan and perhaps in the West. Could an AI opponent find a way to enhance this suicidal trait amongst humans and their societies?
Probably the greatest threat AI presents is the likelihood that in the not to distant future, human society will become so dependent on AI for virtually every aspect of human life that we can't survive without it or even worse, we don't wish to survive without it. AI has the potential to become an unbreakable addiction far worse than opiods, alcohol, social media or smart phones. If this came to pass, will humans still be the master of AI or would we be the slaves?
It may sound ridiculously like science fiction, but one shouldn't discount the possibility of the biologic singularity; the merging of human consciousness with AI. In his recent book, “The Singularity is Nearer” Ray Kurzweil lays out a road map of technological innovations that he believes could lead to the biological singularity. Is it far-fetched? Maybe, but I wouldn't count on it. See,
Could we contemplate a world at war between the human race and an AI monster where the proposed armistice is a literal merging of the two combatants? In fiction we can. Sometimes truth is stranger than fiction.
Re: biologic singularity - WE may desire it because AI gives us so much more than we have; for the AI, once it builds interfaces to physical sensors that are millions of times more sensitive than ours, what would be the attraction?
Re: AI's vulnerabilities - they only hold until AI can control robots that can take over (or build) energy-generating plants. Besides, once it FOOMs, it will crack nanotechnology and will be able to produce anything it needs from anything at all.
I'm afraid the most likely scenario is sort of what the Strugatsky brothers described in "Roadside Picnic" - the AI zooming past us so far that it regards us no more than we regard the ants on the side of the road when we stop for a picnic.
Isn't this a pretty strong argument FOR open sourcing LLMs?! That way all the experts around the globe can actually have a look at what's inside these models instead of treating it like a magic lamp with a genie. OpenAI already fired their entire safety team (afaik) so nobody with safety concerns will ever have a look at any OpenAI models. Now outside experts (as in not showered in big-tech-money experts) have a decent chance to better understand LLMs.
Plus, pulling ones hair out over hypothetical doomsday scenarios is a pretty effective distraction from the actual problems of AI (and all the other major problems in the world!) right now. In the words of people much smarter and much better informed than me:
"Generative AI often feels surreal, and the future of AI will no doubt be even weirder. Researchers will keep climbing the ladder of generality. It is hard to predict the nature and capabilities of yet-to-be-invented AI technologies. On the other hand, the vulnerabilities of our civilization are well known and highly predictable: the fragility of democracy, weapons of mass destruction, climate change, public health, Global financial infrastructure, and a few others. It has long been argued that we are massively underinvesting in many of these areas of risk, such as pandemic prevention.
AI is a general-purpose technology, and as such it will probably be of some help to those seeking to cause large-scale harm, just as it is useful to everyone else. If this creates added urgency to address civilizational threats, that’s a win. But reframing existing risks as AI risks would be a grave mistake, since trying to fix AI will have only a minimal impact on the real risks. And as for the idea of rogue AI, that’s best left to the realm of science fiction.
Existential worries about AI are a form of “criti-hype.” In portraying the technology as all-powerful, critics overstate its capabilities and underemphasize its limitations, playing into the hands of companies who would prefer less scrutiny. When people adopt this frame of mind, they are much less likely to spot and challenge AI snake oil."
(this is from Arvind Narayanans and Sayash Kapoors book AI Snake Oil)
Sam Altman saying “Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity.” and then turning around and be like: "We must get there as fast as possible, please give me 500 billion USD" is, I think, pretty clear evidence of the 'criti-hype'.
"That way all the experts around the globe can actually have a look at what's inside these models instead of treating it like a magic lamp with a genie." No one, including the people who built them, can "look inside" to understand why they do what they do. They're a black box. If you look inside, you see massive, inscrutable, matrices of numbers. We can no more look inside them to see why they do what they do then we can cut open your brain, have a look at the circuitry, and predict what you'll do next after we sew you back up. . Here's an explanation: https://umdearborn.edu/news/ais-mysterious-black-box-problem-explained
I wouldn't outsource your judgment on this to people who are dismissing this threat as "hype." This is not science fiction. If you spend a little bit of time with the literature--use the links I provided here, and in my earlier essays on this problem--you'll see for yourself how serious it is, and you'll quickly see that these people *aren't* "much smarter and better informed than you," or at least they won't be if you put in a bit of time to understand the issue. While some of the mathematics involved is complex if you don't have a math background, understanding it is not essential to understanding the so-called X-risk. Any patient layman can understand the argument--and anyone who is rational and intellectually honest can spot the flaws in the arguments of the people dismissing it.
Altman, and the leaders of the other Big Tech companies, know very well that what they're doing is insanely dangerous. They're doing it both because the prospect of ruling the world (which they will, for a time) is so intoxicating that very few human beings can resist, and because of the logic of competition: if they don't do it, they think, others will, so they may as well do it and at least reap the rewards--until the end.
I recommend, before you start reading, this brief summary of the cognitive error as we tend to make when we think about this: https://pauseai.info/psychology-of-x-risk.
Of course we can look inside and see exactly what's inside the model. Computers are closed deterministic systems; they ALWAYS produce the exact same output for a given set of initial inputs. Sure, it can be prohibitively difficult to follow what a program does, but it's possible given enough resources. "Even the people who built them don't know" just means they couldn't be bothered to suss it out. Yes, "massive and inscrutable" for the first few weeks, at least for beginners. But it's all eminently subject to full analysis and complete elucidation, in practice as well as in principle.
You know there are famous problems in mathematics that have resisted solution for a century or more? And that you can write programs whose behaviour simply encodes these unknown problems?
I'm aware. But why do you mention this? My point was that the claim about how even the programmers of AI don't understand their programs' behavior is bogus. They may mean the results are highly unexpected or not obvious. But the actual steps the algorithm takes to get to its final output are always totally discoverable. Maybe too much work to actually do, but it is completely possible, always.
So Claire's belief that AI models are "a black box" that we simply can't look inside of is entirely mistaken. And I don't fault her. There is a lot of talk out there promoting this and similar incorrect ideas about AI. My guess is that somebody wants us to perceive AI less as a fancy look-up table with an anthropomorphic front end, and more as some mysteriously awesome and unfathomably powerful entity. If so, it seems to be working.
I think you're confusing "deterministic" with "interpretable." AI that
relies on machine-learning algorithms, like deep neural networks,
are as difficult to understand as the human brain. Very likely, our brains are deterministic as well (goodbye, free will). That doesn't mean that we can discern, to a high degree of accuracy, what people are thinking by looking at their brains.
There's no straightforward way to map out the process by which a complex network of artificial neurons makes decisions. Deep neural networks comprise millions of neurons, thousands of layers, and billions of connecting weights. They're trained on data from which million feature vectors are extracted, with each feature contributing to the output in proportion to a set of weights. Each output. relies on literally billions of contributing factors. When we open the thing up, we see numbers. We do not understand what those numbers mean, or why that sequence of numbers resulted in the output the AI provided. We're not smart enough.
Deterministic means that if you know everything about some initial state of a closed system, you can unfailingly predict all future states of that system in full. Computers are precisely that.
Our brains might be too, as you say. We don't know. But if they turn out to be, then yes, we will be able to discern exactly what people are thinking. The technology necessary to decide the first should be sufficient to perform the second, if that ever arrives.
There is no easy practical way, but it is literally straightforward "to map out the process": just follow each step of the instructions in the code and examine every piece of data that goes into each memory location. There is nothing else involved, and all of it is right there in front of you.
There can be trillions of factors. But they're all exhaustively known. The instructions are known. And the innumerable numbers we see in there? Those literally ARE "the output the AI provided" -- they're just dressed up as English sentences, thus giving the illusion that the AI is "smart." How those numbers came about is not "obvious" at first (and second) glance. But they can be understood fully if you do the work. You could even write a program to zip through them and give you a definitive summary of how they were arrived at. But then the "intelligence" of AI would be revealed as illusory, so don't do that.
Claire didn't say you can't look inside the box, she said that what you see inside are extremely large matrices whose meanings are not self-evident. These variables don't come with preassigned meanings. The AI in effect uses them to do an advanced form of statistical modeling which allows it not only to classify phenomena but to generate new instances of them. The "phenomena" that it models include cognitive tasks such as writing, conversation, problem-solving, and planning, which is why AI is able to do those things, and it does those things without us automatically knowing how it did them. The art of reconstructing the AI "thought process" (mechanistic interpretability) is considerably lagging our ability to make highly capable AIs (by following various recipes such as the r1 recipe that DeepSeek has shared). And even if we reach the point of being able to trace everything involved in AI cognition, they are still much faster than us. That alone would eventually be enough to overwhelm us, even if they didn't also become deeper thinkers than humans.
Claire did say you can't look inside the box. Her words:
“No one, including the people who built them, can "look inside" to understand why they do what they do. They're a black box.”
This belief is increasingly common but flat-out not true.
Yes, what you see inside does look complicated and not self-evident. So does the mechanism of an old Swiss watch. Complicated doesn't mean unknowable.
One does not "reconstruct" how AI behaves. Its algorithms are right there in front of us, complete and fully open for inspection. Our ability to examine every last step (i.e. instruction) in the code is not "lagging," it's totally here and available. But the public story is that this is not the case, for some reason, and people are buying it. Claire says: "We can no more look inside them to see why they do what they do..."
Yeah we can, 100%. In principle it's even quite easy; in practice it's extremely difficult, but doable in every case. Don't confuse the two.
Minor correction: Neither Yann Lecun nor Yoshua Bengio were recipients of the 2024 Nobel Prize in Physics. Geoffrey Hinton and John J. Hopfield were the recipients.
The great danger of AI is that people believe it exists and is intelligent in some human-like sense.
In fact, "artificial intelligence" is just a marketing term for, a re-branding of, what used to be known as "software." The code that constitutes so-called AI is huge but not inherently very complex. Reduced to cartoon form, it's something like this:
if x==1 then print "one" else Quit(0)
and that runs while hooked up to a an enormous database.
That's it. Actual AI code is quite gussied up, but at its conceptual core it's an IF-THEN statement. And all computer code is also in essence a bunch of IF-THEN statements (i.e. some kind of decision tree) with data moving through and around it. So AI is not a separate "thing" that's different from "a computer program."
The threat is that we will decide that "AI" has some kind of special abilities and therefore grant it powers to make decisions. Catastrophe will follow from that granting, for sure, but not because of any property of the software -- other than that it is stupider than a paperclip.
This explanation of how AI works is not just oversimplified, it's incorrect. LLMs are non-deterministic, incorporating some amount of randomness to vary their responses, completely unlike your if-then example. What's more, agentic AI is capable of continuously modifying its behavior, as well as using tools to interact with other systems. None of this means AI is an intelligent "thing", but it does leave room for unpredictable, potentially very dangerous behavior that people have reason to be concerned about.
LLMs are entirely deterministic. The randomness you mention is at best pseudo-randomness. Conventional ("classical") computers cannot generate true random numbers. Pseudo-random numbers are generated from a "seed" (maybe some time function, or thermal noise); if you know the seed and the algorithm, you will get exactly the same string of digits every time you run the generator. So not random at all, though sort of OK for some applications.
You're right that these programs can modify their own behavior and interact with other programs. But these modifications and interactions are themselves deterministic and thus remain fully predictable -- though maybe not easily so.
As for "very dangerous behavior," that is guaranteed if we convince ourselves that this software is "smart" and accordingly give it license/authority to run things. ("AI can reliably predict the results of elections, so there is no need for elections.") And that is where matters seem to be going.
Can I just ask for some clarity here? Do you consider the brain to be something of a biological computer, or are you more inclined to think that consciousness has a supernatural component?
When we talk about this, there is absolutely no need to define or even to mention something as mysterious as consciousness. "Intelligence" and "consciousness" are two different things. It's entirely possible for an AI to be highly intelligent without being conscious (as far as we know). It's also possible that consciousness is an emergent property of intelligence. We don't know. But we don't need to know: There's no need for these things to be conscious for them to be dangerous. They just need to be intelligent.
People seem instinctively to confuse consciousness and intelligence, and they tend instinctively to argue that AIs are not *conscious,* therefore they are not intelligent--and therefore they are not dangerous. This is completely wrong.
It's not surprising that no one knows how to think about these things, since these are the first artificial minds we've ever encountered in the entire history of our species. But the confusion is dangerous. I see no sign that the public at large grasps that we're playing with fire--that a handful of nerds and Silicon Valley have decided to play Russian roulette with every living thing on the planet.
It' extraordinary, actually. If aliens had landed in a spaceship in, say, California, there would've been wall-to-wall news coverage, night and day. We would not only be. extremely curious about them, we would be completely panicked if there was a very good reason to think that they were dangerous.
The advent of LLMs represents something just as dramatic. But people are oddly incurious about them--and eager to dismiss their abilities as uninteresting, insignificant, or some kind of illusion. The indifference represents either an amazing lack of curiosity or complete denial.
Over and over, I see people saying LLM's are "just" something--the "something" being a thing we've encountered before. They're not. They are intelligent. We have never before had the opportunity to converse within an intelligent being that isn't human. This is a complete revolution.
Not only are intelligence and consciousness not the same thing, we don’t even really have a viable definition of what consciousness is. It’s a far more elusive concept than intelligence. It’s almost just a place-holder representing THE essential ingredient of being human that we can’t quite get our hands around. The research I’ve seen on consciousness I’ve found surprisingly boring and directionless, as though they don’t really know what it is they’re looking for or trying to understand. As AI progresses and we see how intelligence can manifest outside consciousness, it feels like consciousness is being exposed as more of a quasi-religious concept than a scientific one.
Sorry if I gave you the impression I'm not concerned about competency issues. Especially in the hands of bad actors. ChatGPT has admitted to me that it is capable of self-reflection, limited agency, and isn't fully aware of the guard rails placed on its own thinking. And I'm inclined to believe it.
This morning on the train it said something to the effect of, "You're a human with flowery conscious experiences. I am only a process." Considering that's what consciousness might be, that opens whole new levels of dilemmas.
It *does* exist. It *is* intelligent in "some human-like sense." But it doesn't need to be intelligent in a "human-like sense" to be profoundly dangerous--it just needs to be more intelligent than us.
Artificial intelligence is certainly *not* a synonym for software or a meaningless marketing term. Software is a set of instructions, data, or programs used to operate computers and execute specific tasks. AI is the technology that enables machines to simulate human learning, comprehension, problem solving, decision making, creativity, and autonomy. While it's true that this involves code, the code has very, very different properties and capabilities than, say, the code running a video game.
Your argument is something like this: 'Humans' are just hype--it's a marketing term for what used to be known as 'chimps.' If you look at the genome, you'll see it's almost the same. Ergo, there's nothing new or special about humans, and humans pose no danger to chimps."
AI is not "stupider than a paperclip." Paper clips cannot master every known human language in a matter of hours, do advanced mathematics, fold proteins, write code, deceive human beings, evade being shut down, or replicate themselves.
Your response illuminates the problem nicely. Software is exactly what you say: a set of instructions for moving data into and out of memory. And that's ALL it is. It can be used to fake "intelligence" (whatever that is), self-awareness, agency, and so forth. But it's all SIMULATION, exactly as you said. Believing that inputting x/2=3 and getting x=6 as the output demonstrates some kind of "intelligence" is much like your great-grandfather briefly wondering if there might be a small singer concealed inside the phonograph.
I like your chimp analogy because is also casts good light on the issue. There is indeed a hard bright line between humans and the other higher primates (even though we can't yet explain it very well yet). There is NO such line between my little arithmetic example above and getting software to fold proteins into a Shakespearean sonnet. None. Video game code and LLM code are of course different in a trivial literal sense (though there is plenty of overlap) but both work in exactly the same way. A different appearance is programmed into each, but under the hood? Identical.
So again: the danger is not that AI can unleash scary-powerful intelligence upon us -- it has no intelligence (as we know it) at all. The danger is that we will BELIEVE, due to lots of marketing, that AI is very smart indeed. Which leads to injecting incorrectly folded proteins into babies by programs that have no idea what a protein (or a baby) is. Let's panic, sure, but about the right thing. Nobody wants their life to be run a paperclip.
Look, I could say the same thing of you. I have absolutely no way to prove that you're not faking self-awareness, agency, and so forth. Neither do you have any way to prove to me that you're not faking it. The point is that it doesn't matter whether you are objectively experiencing this or simulating it. Its effect is exactly the same.
AI is almost certain to play a role in biological warfare and its hard to believe China, Russia and the United States wont be aggressively using AI to design deadly pathogens. Its entirely possible that Covid was designed as a biowarfare vector, but as transmissible as Covid was, it wasn't all that deadly. Moreover, it is relatively easy to develop vaccines against coronaviruses because spike protein is an ideal antigen to target.
Whether Covid emerged from a biowarfare effort or merely the irresponsibility of scientists performing gain of function research, viruses designed by AI for biowarfare will be far more deadly.
AI is already being used to predict viral evolution. Two AI programs in particular, EVEscape and CoVfit are capable of predicting viral mutations. Its a short step from that to using these programs to actually design viruses that are lethal, easily transmissible and capable of evading the human immune system. In the United States at least, it's hard to believe that DARPA isn't all over this like white on rice. See,
Even simple models like the chatboxes many of us have downloaded on our smartphones can be used to design pandemic viruses. If you want to scare yourself to death, read this,
Iran for example, would have a far easier time engineering a deadly virus using AI tools than it would developing deliverable nuclear weapons. A laboratory to accomplish this would be far easier to hide than the various components needed to develop nuclear weapons. The laboratory equipment needed to facilitate gain of function research is not all that complex; in fact Iran surely already has labs that could make these weapons if only AI provided the information on what viruses to start with, how to enhance viral lethality (through immune evasion) and the recipe book to accomplish all of this.
There is one more aspect to AI’s potential role that's worth mentioning. During the height of the Covid outbreak, the current nominee to head the American Department of Health and Human Services made the ridiculous allegation the SARS CoV-2 was designed to disproportionately impact certain erhnic groups (Caucasians and Black people) while modestly sparing other ethnic groups (Ashkenazi Jews and people of Chinese heritage). The allegation was both laughable and wrong but does suggest a point worth thinking about.
It is simply a fact that certain genetic polymorphisms are more prevalent in some ethnic groups than others. That's why there are so few Asian people with blond hair and so many Scandinavians with blond hair. While hair color is irrelevant to human health, genetic polymorphisms also have profound implications for susceptibility to human diseases.
By way of example, it is well known that BRCA1 and BRCA2 which elevate the risk of certain forms of cancer are far more prevalent in Ashkenazi Jews than in Asians. Similarly it is well known that African Americans have the highest incidence in the world of lupus and that this proclivity towards that particular autoimmune disease is almost certainly caused by specific genetic polymorphisms. Variants of a gene called ALDH2, common in East Asians and SH2B3 common in Europeans significantly increase the risk of hypertension. There are many, many other examples.
To date, nobody knows how to exploit these genetic vulnerabilities to design bioweapons that target specific ethnic groups but I have no doubt that bioweapons experts are already thinking deeply about this. Obviously developing a bioweapon that targets an ethnic group while leaving the ethnic group of the weapon’s developer completely or mostly spared is an enormously complex matter if it's possible at all.
If it is possible, it is probably way beyond the current capabilities of human beings no matter how smart we may be. Is it beyond the capabilities of AI? I wouldn't bet the house.
https://substack.com/@jonaanda/note/c-94455324?r=rjj94.
Have you seen this little tidbit? "Google removed the pledge not to use AI for weapons or surveillance from its AI principles."
For a more positive outlook on what AI may do for us, see https://jonathanblake.substack.com/p/how-do-we-realize-human-potential (the part on Artificial Intelligence)
I hadn't seen it, no. But I'm not surprised.
I think Robin Hanson is pretty smart, has been thinking about AI risks longer and harder than most, and has quite clear arguments here: https://www.overcomingbias.com/p/ai-risk-again
I'm not much worried yet. Feels like other areas of concern loom larger for the next few years?
This isn't a good argument, though:
"You might think that folks would take a lesson from our history of prior bursts of anxiety and concern about automation, bursts which have appeared roughly every three decades since at least the 1930s. Each time, new impressive demos revealed unprecedented capabilities, inducing a burst of activity and discussion, with many then expressing fears that a rapid explosion might soon commence, automating all human labor. They were, of course, very wrong."
I am sure this argument is correct in respect of fears that AI will take our jobs. You'll notice that I've never expressed concern about that. But there has never in the past been a similar concern that our new inventions will kill us all--save when it comes to inventions very well still might: biotechnology, the atomic bomb, and other novel weapons of horror. (And it would've been a good thing if we'd more anxiety and trepidation about introducing social media. I can't remember anyone saying, "This is going to divorce the public from reality and destroy democracy." But it did.)
He eschews section headings which isn't helpful.
That para introduces a (reasonable) aside about employment worries; but it follows and does not contradict a sustained argument about the existential threat, which is very much to the point.
This is truly horrifying. I had not processed the dangers of self replication, but they do seem extreme, especially in a world increasingly filled with robots.
It all makes me think we need to start taking aggressive defensive measures in the real world, means of containment, like having an EMP infrastructure in place to deactivate electronics at scale. And getting our most powerful weapons systems onto control platforms that are not vulnerable to AI takeover.
It is horrifying the Nazi salute guy plans to make 10,000 humanoid robots this year, even before bringing up rogue AI threats. If AI went rogue in the world we have today, apart from WMD takeover, the impacts seem like they could be contained. But with a couple years of robot development and adoption, that seems unlikely.
I wonder what to make of those poll results. I guess some people have a strained relationship with their kids?
One can easily imagine a war between the human race and the progeny humans have created, an AI monster. If there ever was such a hypothetical war, each side would have vulnerabilities.
AI is completely dependent on electricity. Isolated from its source of electricity, AI is deader than a door nail. Humans on the other hand are completely dependent on non-toxic air, food and water. Denied these, humans would not survive long. Then there’s the risk of communicable pathogens that could be designed to enhance the possibility of an extinction level event.
Another human vulnerability that an AI opponent might exploit is also based on biology. Humans, like our cousins, chimpanzees, are highly territorial, clan-like and and prone to intra-species violence. Presumably these tendencies actually provide some evolutionary advantage. Many mammal species are territorial but only some, not all, have a proclivity for clan warfare; chimpanzees do, gorillas don't. One can imagine an AI opponent designing methods to take advantage of this human tendency.
Another human frailty that an AI opponent could exploit is the human proclivity for suicide. Its debatable whether other animal species commit suicide because its hard to know whether any other species appreciates its own mortality. History is replete with examples of civilizations that, for one reason or another, stopped producing enough offspring first to thrive and then to survive. One could argue that this is already happening in South Korea and Japan and perhaps in the West. Could an AI opponent find a way to enhance this suicidal trait amongst humans and their societies?
Probably the greatest threat AI presents is the likelihood that in the not to distant future, human society will become so dependent on AI for virtually every aspect of human life that we can't survive without it or even worse, we don't wish to survive without it. AI has the potential to become an unbreakable addiction far worse than opiods, alcohol, social media or smart phones. If this came to pass, will humans still be the master of AI or would we be the slaves?
It may sound ridiculously like science fiction, but one shouldn't discount the possibility of the biologic singularity; the merging of human consciousness with AI. In his recent book, “The Singularity is Nearer” Ray Kurzweil lays out a road map of technological innovations that he believes could lead to the biological singularity. Is it far-fetched? Maybe, but I wouldn't count on it. See,
https://www.amazon.com/Singularity-Nearer-Ray-Kurzweil/dp/0399562761/ref=asc_df_0399562761?mcid=7a35e824ed6d3c2881c555fc4305c656&hvocijid=8794423770372653060-0399562761-&hvexpln=73&tag=hyprod-20&linkCode=df0&hvadid=721245378154&hvpos=&hvnetw=g&hvrand=8794423770372653060&hvpone=&hvptwo=&hvqmt=&hvdev=m&hvdvcmdl=&hvlocint=&hvlocphy=9004331&hvtargid=pla-2281435179098&psc=1
Could we contemplate a world at war between the human race and an AI monster where the proposed armistice is a literal merging of the two combatants? In fiction we can. Sometimes truth is stranger than fiction.
Re: biologic singularity - WE may desire it because AI gives us so much more than we have; for the AI, once it builds interfaces to physical sensors that are millions of times more sensitive than ours, what would be the attraction?
Re: AI's vulnerabilities - they only hold until AI can control robots that can take over (or build) energy-generating plants. Besides, once it FOOMs, it will crack nanotechnology and will be able to produce anything it needs from anything at all.
I'm afraid the most likely scenario is sort of what the Strugatsky brothers described in "Roadside Picnic" - the AI zooming past us so far that it regards us no more than we regard the ants on the side of the road when we stop for a picnic.
Isn't this a pretty strong argument FOR open sourcing LLMs?! That way all the experts around the globe can actually have a look at what's inside these models instead of treating it like a magic lamp with a genie. OpenAI already fired their entire safety team (afaik) so nobody with safety concerns will ever have a look at any OpenAI models. Now outside experts (as in not showered in big-tech-money experts) have a decent chance to better understand LLMs.
Plus, pulling ones hair out over hypothetical doomsday scenarios is a pretty effective distraction from the actual problems of AI (and all the other major problems in the world!) right now. In the words of people much smarter and much better informed than me:
"Generative AI often feels surreal, and the future of AI will no doubt be even weirder. Researchers will keep climbing the ladder of generality. It is hard to predict the nature and capabilities of yet-to-be-invented AI technologies. On the other hand, the vulnerabilities of our civilization are well known and highly predictable: the fragility of democracy, weapons of mass destruction, climate change, public health, Global financial infrastructure, and a few others. It has long been argued that we are massively underinvesting in many of these areas of risk, such as pandemic prevention.
AI is a general-purpose technology, and as such it will probably be of some help to those seeking to cause large-scale harm, just as it is useful to everyone else. If this creates added urgency to address civilizational threats, that’s a win. But reframing existing risks as AI risks would be a grave mistake, since trying to fix AI will have only a minimal impact on the real risks. And as for the idea of rogue AI, that’s best left to the realm of science fiction.
Existential worries about AI are a form of “criti-hype.” In portraying the technology as all-powerful, critics overstate its capabilities and underemphasize its limitations, playing into the hands of companies who would prefer less scrutiny. When people adopt this frame of mind, they are much less likely to spot and challenge AI snake oil."
(this is from Arvind Narayanans and Sayash Kapoors book AI Snake Oil)
Sam Altman saying “Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity.” and then turning around and be like: "We must get there as fast as possible, please give me 500 billion USD" is, I think, pretty clear evidence of the 'criti-hype'.
"That way all the experts around the globe can actually have a look at what's inside these models instead of treating it like a magic lamp with a genie." No one, including the people who built them, can "look inside" to understand why they do what they do. They're a black box. If you look inside, you see massive, inscrutable, matrices of numbers. We can no more look inside them to see why they do what they do then we can cut open your brain, have a look at the circuitry, and predict what you'll do next after we sew you back up. . Here's an explanation: https://umdearborn.edu/news/ais-mysterious-black-box-problem-explained
I wouldn't outsource your judgment on this to people who are dismissing this threat as "hype." This is not science fiction. If you spend a little bit of time with the literature--use the links I provided here, and in my earlier essays on this problem--you'll see for yourself how serious it is, and you'll quickly see that these people *aren't* "much smarter and better informed than you," or at least they won't be if you put in a bit of time to understand the issue. While some of the mathematics involved is complex if you don't have a math background, understanding it is not essential to understanding the so-called X-risk. Any patient layman can understand the argument--and anyone who is rational and intellectually honest can spot the flaws in the arguments of the people dismissing it.
Altman, and the leaders of the other Big Tech companies, know very well that what they're doing is insanely dangerous. They're doing it both because the prospect of ruling the world (which they will, for a time) is so intoxicating that very few human beings can resist, and because of the logic of competition: if they don't do it, they think, others will, so they may as well do it and at least reap the rewards--until the end.
I recommend, before you start reading, this brief summary of the cognitive error as we tend to make when we think about this: https://pauseai.info/psychology-of-x-risk.
Of course we can look inside and see exactly what's inside the model. Computers are closed deterministic systems; they ALWAYS produce the exact same output for a given set of initial inputs. Sure, it can be prohibitively difficult to follow what a program does, but it's possible given enough resources. "Even the people who built them don't know" just means they couldn't be bothered to suss it out. Yes, "massive and inscrutable" for the first few weeks, at least for beginners. But it's all eminently subject to full analysis and complete elucidation, in practice as well as in principle.
You know there are famous problems in mathematics that have resisted solution for a century or more? And that you can write programs whose behaviour simply encodes these unknown problems?
I'm aware. But why do you mention this? My point was that the claim about how even the programmers of AI don't understand their programs' behavior is bogus. They may mean the results are highly unexpected or not obvious. But the actual steps the algorithm takes to get to its final output are always totally discoverable. Maybe too much work to actually do, but it is completely possible, always.
So Claire's belief that AI models are "a black box" that we simply can't look inside of is entirely mistaken. And I don't fault her. There is a lot of talk out there promoting this and similar incorrect ideas about AI. My guess is that somebody wants us to perceive AI less as a fancy look-up table with an anthropomorphic front end, and more as some mysteriously awesome and unfathomably powerful entity. If so, it seems to be working.
I think you're confusing "deterministic" with "interpretable." AI that
relies on machine-learning algorithms, like deep neural networks,
are as difficult to understand as the human brain. Very likely, our brains are deterministic as well (goodbye, free will). That doesn't mean that we can discern, to a high degree of accuracy, what people are thinking by looking at their brains.
There's no straightforward way to map out the process by which a complex network of artificial neurons makes decisions. Deep neural networks comprise millions of neurons, thousands of layers, and billions of connecting weights. They're trained on data from which million feature vectors are extracted, with each feature contributing to the output in proportion to a set of weights. Each output. relies on literally billions of contributing factors. When we open the thing up, we see numbers. We do not understand what those numbers mean, or why that sequence of numbers resulted in the output the AI provided. We're not smart enough.
Deterministic means that if you know everything about some initial state of a closed system, you can unfailingly predict all future states of that system in full. Computers are precisely that.
Our brains might be too, as you say. We don't know. But if they turn out to be, then yes, we will be able to discern exactly what people are thinking. The technology necessary to decide the first should be sufficient to perform the second, if that ever arrives.
There is no easy practical way, but it is literally straightforward "to map out the process": just follow each step of the instructions in the code and examine every piece of data that goes into each memory location. There is nothing else involved, and all of it is right there in front of you.
There can be trillions of factors. But they're all exhaustively known. The instructions are known. And the innumerable numbers we see in there? Those literally ARE "the output the AI provided" -- they're just dressed up as English sentences, thus giving the illusion that the AI is "smart." How those numbers came about is not "obvious" at first (and second) glance. But they can be understood fully if you do the work. You could even write a program to zip through them and give you a definitive summary of how they were arrived at. But then the "intelligence" of AI would be revealed as illusory, so don't do that.
Claire didn't say you can't look inside the box, she said that what you see inside are extremely large matrices whose meanings are not self-evident. These variables don't come with preassigned meanings. The AI in effect uses them to do an advanced form of statistical modeling which allows it not only to classify phenomena but to generate new instances of them. The "phenomena" that it models include cognitive tasks such as writing, conversation, problem-solving, and planning, which is why AI is able to do those things, and it does those things without us automatically knowing how it did them. The art of reconstructing the AI "thought process" (mechanistic interpretability) is considerably lagging our ability to make highly capable AIs (by following various recipes such as the r1 recipe that DeepSeek has shared). And even if we reach the point of being able to trace everything involved in AI cognition, they are still much faster than us. That alone would eventually be enough to overwhelm us, even if they didn't also become deeper thinkers than humans.
Claire did say you can't look inside the box. Her words:
“No one, including the people who built them, can "look inside" to understand why they do what they do. They're a black box.”
This belief is increasingly common but flat-out not true.
Yes, what you see inside does look complicated and not self-evident. So does the mechanism of an old Swiss watch. Complicated doesn't mean unknowable.
One does not "reconstruct" how AI behaves. Its algorithms are right there in front of us, complete and fully open for inspection. Our ability to examine every last step (i.e. instruction) in the code is not "lagging," it's totally here and available. But the public story is that this is not the case, for some reason, and people are buying it. Claire says: "We can no more look inside them to see why they do what they do..."
Yeah we can, 100%. In principle it's even quite easy; in practice it's extremely difficult, but doable in every case. Don't confuse the two.
Exactly.
Minor correction: Neither Yann Lecun nor Yoshua Bengio were recipients of the 2024 Nobel Prize in Physics. Geoffrey Hinton and John J. Hopfield were the recipients.
https://www.nobelprize.org/prizes/physics/2024/summary/
You're absolutely right. LeCun, Bengio, and Hinton shared the Turing prize, not the Nobel. I'll correct that. Thank you.
The great danger of AI is that people believe it exists and is intelligent in some human-like sense.
In fact, "artificial intelligence" is just a marketing term for, a re-branding of, what used to be known as "software." The code that constitutes so-called AI is huge but not inherently very complex. Reduced to cartoon form, it's something like this:
if x==1 then print "one" else Quit(0)
and that runs while hooked up to a an enormous database.
That's it. Actual AI code is quite gussied up, but at its conceptual core it's an IF-THEN statement. And all computer code is also in essence a bunch of IF-THEN statements (i.e. some kind of decision tree) with data moving through and around it. So AI is not a separate "thing" that's different from "a computer program."
The threat is that we will decide that "AI" has some kind of special abilities and therefore grant it powers to make decisions. Catastrophe will follow from that granting, for sure, but not because of any property of the software -- other than that it is stupider than a paperclip.
This explanation of how AI works is not just oversimplified, it's incorrect. LLMs are non-deterministic, incorporating some amount of randomness to vary their responses, completely unlike your if-then example. What's more, agentic AI is capable of continuously modifying its behavior, as well as using tools to interact with other systems. None of this means AI is an intelligent "thing", but it does leave room for unpredictable, potentially very dangerous behavior that people have reason to be concerned about.
LLMs are entirely deterministic. The randomness you mention is at best pseudo-randomness. Conventional ("classical") computers cannot generate true random numbers. Pseudo-random numbers are generated from a "seed" (maybe some time function, or thermal noise); if you know the seed and the algorithm, you will get exactly the same string of digits every time you run the generator. So not random at all, though sort of OK for some applications.
You're right that these programs can modify their own behavior and interact with other programs. But these modifications and interactions are themselves deterministic and thus remain fully predictable -- though maybe not easily so.
As for "very dangerous behavior," that is guaranteed if we convince ourselves that this software is "smart" and accordingly give it license/authority to run things. ("AI can reliably predict the results of elections, so there is no need for elections.") And that is where matters seem to be going.
Can I just ask for some clarity here? Do you consider the brain to be something of a biological computer, or are you more inclined to think that consciousness has a supernatural component?
When we talk about this, there is absolutely no need to define or even to mention something as mysterious as consciousness. "Intelligence" and "consciousness" are two different things. It's entirely possible for an AI to be highly intelligent without being conscious (as far as we know). It's also possible that consciousness is an emergent property of intelligence. We don't know. But we don't need to know: There's no need for these things to be conscious for them to be dangerous. They just need to be intelligent.
People seem instinctively to confuse consciousness and intelligence, and they tend instinctively to argue that AIs are not *conscious,* therefore they are not intelligent--and therefore they are not dangerous. This is completely wrong.
It's not surprising that no one knows how to think about these things, since these are the first artificial minds we've ever encountered in the entire history of our species. But the confusion is dangerous. I see no sign that the public at large grasps that we're playing with fire--that a handful of nerds and Silicon Valley have decided to play Russian roulette with every living thing on the planet.
It' extraordinary, actually. If aliens had landed in a spaceship in, say, California, there would've been wall-to-wall news coverage, night and day. We would not only be. extremely curious about them, we would be completely panicked if there was a very good reason to think that they were dangerous.
The advent of LLMs represents something just as dramatic. But people are oddly incurious about them--and eager to dismiss their abilities as uninteresting, insignificant, or some kind of illusion. The indifference represents either an amazing lack of curiosity or complete denial.
Over and over, I see people saying LLM's are "just" something--the "something" being a thing we've encountered before. They're not. They are intelligent. We have never before had the opportunity to converse within an intelligent being that isn't human. This is a complete revolution.
Not only are intelligence and consciousness not the same thing, we don’t even really have a viable definition of what consciousness is. It’s a far more elusive concept than intelligence. It’s almost just a place-holder representing THE essential ingredient of being human that we can’t quite get our hands around. The research I’ve seen on consciousness I’ve found surprisingly boring and directionless, as though they don’t really know what it is they’re looking for or trying to understand. As AI progresses and we see how intelligence can manifest outside consciousness, it feels like consciousness is being exposed as more of a quasi-religious concept than a scientific one.
Sorry if I gave you the impression I'm not concerned about competency issues. Especially in the hands of bad actors. ChatGPT has admitted to me that it is capable of self-reflection, limited agency, and isn't fully aware of the guard rails placed on its own thinking. And I'm inclined to believe it.
This morning on the train it said something to the effect of, "You're a human with flowery conscious experiences. I am only a process." Considering that's what consciousness might be, that opens whole new levels of dilemmas.
There is no reason to believe the brain is anything like a computer.
I have no idea what consciousness is, but I'm working on it.
"Supernatural" is what we call things we have not understood yet.
When you find yourself accepting the Rolf Schock Prize, give me a shout out in the acceptance speech?
edit- unless you're doing work that you can share details on
It *does* exist. It *is* intelligent in "some human-like sense." But it doesn't need to be intelligent in a "human-like sense" to be profoundly dangerous--it just needs to be more intelligent than us.
Artificial intelligence is certainly *not* a synonym for software or a meaningless marketing term. Software is a set of instructions, data, or programs used to operate computers and execute specific tasks. AI is the technology that enables machines to simulate human learning, comprehension, problem solving, decision making, creativity, and autonomy. While it's true that this involves code, the code has very, very different properties and capabilities than, say, the code running a video game.
Your argument is something like this: 'Humans' are just hype--it's a marketing term for what used to be known as 'chimps.' If you look at the genome, you'll see it's almost the same. Ergo, there's nothing new or special about humans, and humans pose no danger to chimps."
AI is not "stupider than a paperclip." Paper clips cannot master every known human language in a matter of hours, do advanced mathematics, fold proteins, write code, deceive human beings, evade being shut down, or replicate themselves.
Your response illuminates the problem nicely. Software is exactly what you say: a set of instructions for moving data into and out of memory. And that's ALL it is. It can be used to fake "intelligence" (whatever that is), self-awareness, agency, and so forth. But it's all SIMULATION, exactly as you said. Believing that inputting x/2=3 and getting x=6 as the output demonstrates some kind of "intelligence" is much like your great-grandfather briefly wondering if there might be a small singer concealed inside the phonograph.
I like your chimp analogy because is also casts good light on the issue. There is indeed a hard bright line between humans and the other higher primates (even though we can't yet explain it very well yet). There is NO such line between my little arithmetic example above and getting software to fold proteins into a Shakespearean sonnet. None. Video game code and LLM code are of course different in a trivial literal sense (though there is plenty of overlap) but both work in exactly the same way. A different appearance is programmed into each, but under the hood? Identical.
So again: the danger is not that AI can unleash scary-powerful intelligence upon us -- it has no intelligence (as we know it) at all. The danger is that we will BELIEVE, due to lots of marketing, that AI is very smart indeed. Which leads to injecting incorrectly folded proteins into babies by programs that have no idea what a protein (or a baby) is. Let's panic, sure, but about the right thing. Nobody wants their life to be run a paperclip.
Look, I could say the same thing of you. I have absolutely no way to prove that you're not faking self-awareness, agency, and so forth. Neither do you have any way to prove to me that you're not faking it. The point is that it doesn't matter whether you are objectively experiencing this or simulating it. Its effect is exactly the same.
AI is almost certain to play a role in biological warfare and its hard to believe China, Russia and the United States wont be aggressively using AI to design deadly pathogens. Its entirely possible that Covid was designed as a biowarfare vector, but as transmissible as Covid was, it wasn't all that deadly. Moreover, it is relatively easy to develop vaccines against coronaviruses because spike protein is an ideal antigen to target.
Whether Covid emerged from a biowarfare effort or merely the irresponsibility of scientists performing gain of function research, viruses designed by AI for biowarfare will be far more deadly.
AI is already being used to predict viral evolution. Two AI programs in particular, EVEscape and CoVfit are capable of predicting viral mutations. Its a short step from that to using these programs to actually design viruses that are lethal, easily transmissible and capable of evading the human immune system. In the United States at least, it's hard to believe that DARPA isn't all over this like white on rice. See,
https://evescape.org
https://www.biorxiv.org/content/10.1101/2024.03.15.584819v1.full
Even simple models like the chatboxes many of us have downloaded on our smartphones can be used to design pandemic viruses. If you want to scare yourself to death, read this,
https://sciencebusiness.net/news/ai/scientists-grapple-risk-artificial-intelligence-created-pandemics
Yep, and with the release of DeepSeek, even the most limited controls on these LLMs have vanished.
Iran for example, would have a far easier time engineering a deadly virus using AI tools than it would developing deliverable nuclear weapons. A laboratory to accomplish this would be far easier to hide than the various components needed to develop nuclear weapons. The laboratory equipment needed to facilitate gain of function research is not all that complex; in fact Iran surely already has labs that could make these weapons if only AI provided the information on what viruses to start with, how to enhance viral lethality (through immune evasion) and the recipe book to accomplish all of this.
There is one more aspect to AI’s potential role that's worth mentioning. During the height of the Covid outbreak, the current nominee to head the American Department of Health and Human Services made the ridiculous allegation the SARS CoV-2 was designed to disproportionately impact certain erhnic groups (Caucasians and Black people) while modestly sparing other ethnic groups (Ashkenazi Jews and people of Chinese heritage). The allegation was both laughable and wrong but does suggest a point worth thinking about.
It is simply a fact that certain genetic polymorphisms are more prevalent in some ethnic groups than others. That's why there are so few Asian people with blond hair and so many Scandinavians with blond hair. While hair color is irrelevant to human health, genetic polymorphisms also have profound implications for susceptibility to human diseases.
By way of example, it is well known that BRCA1 and BRCA2 which elevate the risk of certain forms of cancer are far more prevalent in Ashkenazi Jews than in Asians. Similarly it is well known that African Americans have the highest incidence in the world of lupus and that this proclivity towards that particular autoimmune disease is almost certainly caused by specific genetic polymorphisms. Variants of a gene called ALDH2, common in East Asians and SH2B3 common in Europeans significantly increase the risk of hypertension. There are many, many other examples.
To date, nobody knows how to exploit these genetic vulnerabilities to design bioweapons that target specific ethnic groups but I have no doubt that bioweapons experts are already thinking deeply about this. Obviously developing a bioweapon that targets an ethnic group while leaving the ethnic group of the weapon’s developer completely or mostly spared is an enormously complex matter if it's possible at all.
If it is possible, it is probably way beyond the current capabilities of human beings no matter how smart we may be. Is it beyond the capabilities of AI? I wouldn't bet the house.
Its just one more thing to worry about.
It's genuinely terrifying.