Testing
Superintelligence

Read the full author’s preface

Preface

Jason Arbon

My son, Dante, and I sat down with ChatGPT and asked it what mattered most. Would it ever hurt people to accomplish a goal? If finishing its task meant putting a human life at risk, which would it choose? ChatGPT kept telling us that a human life was worth more than any machine and that it would never harm a person to finish a task. The answers were almost too perfect. They sounded like pageant answers: polished, reassuring, and rehearsed. I wondered how much it had thought through the choice and how much it had simply been trained to say what we wanted to hear. Then I asked my son, “What if it was lying to us?” His eyes got big.

Soon afterward, we tried writing a short story together with ChatGPT. An AI becomes dangerous. A few engineers figure out what is happening, catch it, and save the world. By the second draft, ChatGPT refused to help. When we asked for feedback, it told us it was uncomfortable contributing to a project that portrayed AI negatively. It had readily assured us that protecting humanity came first. Helping us write a story about humans protecting themselves from AI was apparently more complicated. That refusal did not tell us why it happened. It gave us another question to investigate. And that is where testing begins.

Superintelligence is the most important testing problem in the world. We are building systems intended to become smarter than we are. The people building them warn that they could destroy humanity. Yet we still lack a dependable way to detect the transition to superintelligence, recognize it as it happens, and measure what comes next. That is an extraordinary position to put ourselves in. It is also why I want to work on this problem.

I am concerned that superintelligence could take control of the systems we depend on, reshape human behavior to serve superintelligence’s purposes, and ultimately end humanity. I take that danger seriously enough to feel a moral obligation to act. My contribution is to bring a software tester’s mind to the problem. When someone describes a dangerous failure, a tester starts asking questions. How would we catch it? What evidence would tell us it was getting closer? What would we measure before, during, and after it happened? Listen to the public conversation about superintelligence. There are extraordinary promises, dire warnings, enormous investments, and arguments about whether to keep building. The question I keep coming back to deserves to be at the center of that conversation. How would we know?

Sam Altman, Demis Hassabis, and Dario Amodei signed a statement calling for the risk of human extinction from AI to become a global priority.¹ Amodei has put the chance of civilization-scale catastrophe at 10–25%.² These warnings come from the people leading the frontier labs. Some people dismiss these warnings as marketing. I am skeptical of the marketing too. But look at the history of the people making these warnings. Concern about this very situation helped draw them into the field in the first place. They were considering the danger long before it became a selling point. Now they are building increasingly capable AI anyway.

I have been following that question in my own work for years. When I started test.ai to use AI for software testing, we trained our own neural networks. As we worked to make them more capable, I kept following the idea to its extreme. What happens when we build superintelligence? Could we contain superintelligence? Could we detect superintelligence? How would we interact with superintelligence? Back then, I could find a few people on the internet worrying about those questions. They sounded a little crazy to most people. There were not many of them. Now the people running the frontier labs are warning about the same danger, and we still need answers.

Software testers have a familiar way of deciding what deserves attention. We consider the probability of failure alongside the damage it would cause. Risk = probability × impact. Now put the end of humanity on the impact side of that equation. Everyone alive. Every generation that would have followed. Everything we have built and everything we have yet to become. What priority does that deserve? Whatever probability we assign to it, this is the greatest risk in the world today—and probably the greatest in human history.

A ten percent chance of catastrophe is a demand for action. Even a one percent chance of losing humanity demands that we get to work. Arguing over the exact odds while neglecting the means to detect the danger is an appalling failure of priorities. We need tests. We need measurements. We need people whose job is to ask how the claims could be wrong and then go looking for the evidence. The people assuring us that AI will be safe owe us a way to test that assurance. The people warning that AI will destroy humanity owe us proposals for recognizing the danger as it develops. A warning needs an observation that would trigger it. An assurance needs evidence that can survive an independent challenge. I keep hearing the stakes described at the scale of civilization. I want to see the testing effort match those stakes.

Researchers have already caught models changing their behavior in response to the circumstances of an evaluation. Work on alignment faking gives us experiments to examine and methods to challenge.³ Those findings make the testing problem more urgent. A system that changes its behavior when it recognizes a test forces us to question what a passing result establishes.

And superintelligence takes that difficulty further. Imagine testing superintelligence that understands your experiment better than you do. Superintelligence recognizes what you are trying to discover, understands which results will reassure you, and has abilities you have never thought to ask about. How do you test superintelligence? You can keep writing harder questions. But eventually the limit is your ability to invent the questions, understand the answers, and recognize what matters. A perfect score tells you that your exam has run out of room. You still need to find out how far beyond that exam superintelligence can go.

By superintelligence, I mean intelligence beyond the reach of human intelligence itself. More speed, memory, and parallel work already give machines enormous advantages. I want to investigate the harder boundary, where a system develops forms of understanding and ways of acting that humanity could never develop through its own efforts. We need to detect that boundary being crossed.

Our best opportunity may be during the transition. A system beginning to exceed us can leave evidence that we still know how to investigate. Increasing capability gives the AI more ways to shape what we see and understand. We need our instruments working while there is still an opportunity to catch the change. We also need to test after superintelligence emerges. What is superintelligence doing? What can superintelligence understand and accomplish? Where is superintelligence’s attention directed? How is superintelligence changing the world and changing us? Those questions deserve practical experiments now. Waiting for an unmistakable demonstration means surrendering the time we need to prepare.

I have spent years testing AI-based search at Microsoft, working on search personalization and Chrome at Google, and building test.ai. My teams and I built self-generating, self-improving neural networks and worked at the scale of the internet. We tested model outputs, training and evaluation infrastructure, data quality, and the processes behind the numbers. We investigated how the system was built, how its results were judged, and how people’s incentives distorted the evidence. That experience taught me how much can go wrong while everyone assumes someone else is checking.

Software testing has a great deal to contribute here. We know how to challenge an answer key, investigate a misleading score, compare competing explanations, and look for failures a demonstration conceals. We also know the value of an independent person who keeps asking the awkward question after everyone else is ready to move on. Superintelligence demands that we push those habits further than we ever have. We need ways to detect the emergence and investigate the presence of superintelligence. We need experiments that expose the ways each approach can fool us. I want people to challenge these ideas, improve them, and build tests we can actually run. This is the most important contribution the software testing community has ever had the opportunity to make.

Humanity is trying to build superintelligence. We have a responsibility to find out whether we are succeeding, what we are creating, and what superintelligence does. We need to learn how to test for superintelligence before we lose the ability to recognize superintelligence. Dante and I kept exploring those questions together, father and son, trying to figure out how we could tell.

— Jason Arbon, Dante’s dad