Testing
Superintelligence

Read the full author’s preface

Preface

We Need More Than a Warning · Jason Arbon

My fourteen-year-old son and I tried asking ChatGPT questions that might tell us whether it was smarter than we were. Some of the answers sounded like pageant answers: polished, reassuring, and difficult to learn much from. Then I asked him, “What if it was lying to us? How could we detect that?”

His eyes got big. The same answer could come from a system trying to mislead us or from one doing what it had learned to do: give a reassuring answer. What could we test that would help us tell the difference?

This book is about testing for superintelligence. When an AI system appears to do something beyond human capability, how do we check the claim, and how would we notice if those results started to accumulate across fields? Progress may hit limits, and capable systems may remain under effective control. In either case, we need ways to find out what they can do while we still have time to act on what we learn.

I mean superintelligence in a strict sense. Imagine the best human teams with more time to learn, invent tools, and work together. Could they acquire the underlying competence? Faster work, more parallel attempts, and even victory over every expert we can recruit leave that question open. The hard case is a system that repeatedly develops and uses competencies beyond what finite human effort could attain under stated conditions. That possibility is the subject of the book. The failures of current systems matter here when they show how we might miss or misread evidence of such a change.

A note after the 2026 U.S. midterms

November 5, 2026. On November 4, following the U.S. midterm elections, Representative Yassamin Ansari called for a pause on advanced AI development and “a ban on artificial superintelligence,” as reported by Tom’s Hardware. The idea was already before Congress. On September 23, Representative Greg Casar and Senator Bernie Sanders introduced the Ban Artificial Superintelligence Act. “Our bill bans the development of artificial superintelligence,” Casar said. The proposal would also prohibit deployment and create an agency to monitor frontier systems. [121, 122]

How do you enforce a ban on something you cannot reliably detect? A rule can target observable conduct, such as running a covered training job without authorization. Records can help establish whether that conduct occurred. A rule keyed to a system’s capabilities poses another question. What did the system actually become capable of, and when? Evidence of a deliberate attempt, evidence that a threshold was crossed, and evidence that a particular person or organization violated a rule answer different questions.

A team could cross a capability threshold while pursuing a narrower objective. A system might also cross it before anyone has a test sensitive enough to notice. Whether either event has happened remains open. That uncertainty matters today. If lawmakers draw a boundary, investigators need to know what they can observe, what they can verify independently, and where their tests leave them blind. The proposed bill’s definition and this book’s stricter human reachability boundary serve different purposes. Any test has to say which claim it is trying to establish.

Why a software tester is asking

I have spent years working on testing at scale, testing AI, and using AI to test software. I was a test manager on AI-based search systems at Microsoft, in the work that became Bing, and on Chrome at Google. These problems were part of my working life long before the current enthusiasm for large language models.

In my recollection of the search competition, Google seemed to have all the smart engineers. Neural networks felt like a practical way for us to compete: let a learned system find improvements we could not keep handcrafting. What felt like a hack became more capable over time. For those of us testing it, that meant learning how to examine a system whose behavior was not simply a list of rules we had written.

That responsibility extended beyond examining model outputs. My team was also accountable for the infrastructure used to train models, evaluate them, and run evaluations again. We examined statistics about the training data and how effectively people were using that infrastructure. We did basic analysis of training configurations and model hyperparameters to understand whether the choices people were making were helping. Testing meant looking at the machinery and the work practices that produced the result, not only the result at the end.

We also worked on data quality, including the human judgments used for training. In one case I remember, a vendor responded to disagreement between raters by firing one of them. That practice was distorting our training data. I return to it in the testing-foundations chapter because it captures a larger problem: apparent agreement can improve while the process producing it becomes less trustworthy.

I later cofounded test.ai, before the current LLM wave, to apply AI to software testing. Google's AI-focused venture fund, Gradient Ventures, led its 2018 Series A. [26] Testing AI and building AI that tests have been connected problems throughout my work.

So yes, I sometimes find myself arguing with a podcast or a conference presentation. I hear a familiar testing failure presented as a new mystery: trusting an answer key nobody checked, confusing a score with coverage, or accepting a demonstration as evidence of reliability. Software testers have spent years encountering these patterns, learning which techniques help, and finding out where they fail.

We have also spent years thinking about oracles: how to judge a result when the correct answer is difficult to specify. A research benchmark and a production testing program face different demands. Production adds changing environments, messy inputs, integrations, users, and failures that escape the test environment. I want more of that experience in this conversation.

This is not a claim that frontier researchers never test or that existing testing practice solves superintelligence. It is a request to bring the accumulated experience of software testing into the work, then discover precisely where it stops being enough. I do not hear enough public discussion of this less glamorous work: testing the training infrastructure, examining data quality, and investigating the incentives behind the labels. That does not establish how much of it happens inside labs. It does mean I want to see the evidence, rather than assume someone has covered it.

At a testing conference, I brought up the problem between sessions. We agreed it sounded hard. A few minutes later we were back to talking about prompts for testing websites. I left wanting a concrete experiment that testers and researchers could pick apart together.

A warning needs a test

I do not want our testing strategy to depend on everybody agreeing to stop. That is a planning position, not a prediction that policy cannot change anything. Pauses, restrictions, and technical controls may matter. We still need methods that work while capable systems exist and people continue trying to use them.

A sandbox is not an answer to all those questions. It is something else to test. An air gap removes important connections, but it does not establish that every physical or human communication channel is absent. A model that helps build its own protections has not independently certified them. And a promise that a system will behave safely is an output to examine, not a reason to stop examining.

We should take those concerns seriously without pretending we have proved every boundary will fail. The practical question is what a control prevents, under which assumptions, and what evidence would reveal its failure. This book will return to that question often.

The test is allowed to be wrong

We build an exam, give it to a model, and debate the percentage. Meanwhile the questions may be ambiguous, the reference answers may be wrong, the scoring software may have bugs, and the environment may be helping the model in ways we did not intend. Documented benchmark reviews show why these are real testing questions. [3, 4]

Some evaluations go far beyond multiple-choice questions. That is progress. But complexity does not make an evaluation trustworthy. We must investigate how it works and how far its conclusions reach.

Now add a more difficult possibility: the system gives us an answer we could not have created. Sometimes we can still check it. Sometimes checking it is itself a research project. Sometimes we may not even recognize the question we should have asked.

Suppose an AI produces a mathematical proof that an expert team could not find. That is a Super Signal, an observation worth investigating. A Super Test starts by recording the claim and comparing the time, tools, and prior work available to the model and to the experts. Could the model have seen the proof before? Ask an independent team to check the proof and then test the system on a fresh problem. Repeat under comparable conditions to see whether the advantage persists. A valid proof settles whether the result follows. The rest of the investigation tells us how much we learned about the system that produced it.

Begin with today

After the opening allegory, Chapter 1 walks through a fictional Super Test. The first hypothesis gets weaker as we improve the baseline, and fresh problems reveal a narrower capability. Chapter 2 takes the same habits to a public benchmark record. Later chapters extend the investigation to learning beyond human examples and to systems that act in laboratories, robots and simulated worlds.

A fictional research assistant connects the chapters. It passes an exam, produces an unfamiliar result, asks for tools, acts, and eventually challenges our ability to judge and observe it. This is a teaching scenario, not a claim that one current product already combines every capability described here.

If we wait until a system is unmistakably beyond us to invent the tests and instruments, the time we need to prepare may already be gone. We should start while we can still discover our mistakes in smaller, more recoverable settings.

Two sides of the nest

The owl and sparrows return throughout the book in short illustrated boxes. “From the sparrows’ side” asks what observers with limited access can see, check, or change. “From the owl’s side” offers a possible explanation for the same scene and suggests what evidence could distinguish it from another.

If a broken latch works again by morning, the sparrows know it was restored. The owl may have noticed the break or replaced the latch on schedule. A comparison with an untouched latch helps the flock tell those accounts apart. Sometimes the owl helps, sometimes it misunderstands, and sometimes it changes its behavior under inspection.

Why the owl

The owl is a deliberate homage to Nick Bostrom's Superintelligence: Paths, Dangers, Strategies. His book explores routes to superintelligence, takeoff dynamics, and the control problem. It also discusses monitoring, boxing, and tripwires; it would be unfair to say it ignores detection and control. [24]

The dedicated deception and safety sections also examine researchers who are already working on these problems.

The question I want to bring into the foreground is the working tester's question: how do we recognize a consequential capability, challenge the claim, and know whether our observation and control mechanisms are working? The homage is an acknowledgment of that earlier work, not an endorsement or collaboration.

This is a developing technical book. Every chapter is a working draft. Executed examples are distinguished from proposed experiments. As the work grows, claims should acquire better evidence, examples should become reproducible, and weak arguments should change. A book about testing ought to make it possible to test what it says.