Testing
Superintelligence

Question 1

Would superintelligence know all the answers?

After our conversation with ChatGPT, Dante wanted to see what a harder question would reveal. I was about to suggest a harder question when he leaned forward. “Would superintelligence know all the answers? Then what do we ask?”

“Suppose the AI gets every question right,” I said. “What score would the AI get with twice the knowledge?”

He thought for a moment. “The AI would still get 100%, even with twice the knowledge.”

That is what bothers me about a perfect score. The exam has run out of room, and we are still looking at the number as though it tells us how smart the AI is. We also have to check the exam itself. Were the questions any good? Was the answer key right? Could the AI have seen the answers before?

“What if we ask something nobody knows?” Dante said. Then he stopped. “But how would we even know what to ask? What if the AI knows things we don’t even know to ask?”

I gave him an imaginary case. An AI finds a mathematical proof that a team of mathematicians couldn’t find. Someone outside the project checks every step. The proof holds. Then the team finds an older result that does much of the work. The answer is still right, but now we have another explanation for how the AI got there.

“Then what do we ask?” I said.

We could give the AI three fresh problems that need the same idea. Suppose the AI proves two. On the third, we change something the method depends on: the original problem was symmetrical, and this one isn’t. The AI’s proof fails. Another tester tries the changed problem and gets another invalid proof.

“So the AI doesn’t know all the answers.”

“We found one the AI couldn’t give us a valid proof for,” I said. “And we found something about where the AI’s method breaks.”

That tells me more than the perfect score did. We have two proofs someone else checked and a failure we can investigate. I still want to know how far the AI can go. Superintelligence is a much bigger claim, and a one hundred percent score cannot answer Dante’s question for us.

“What if AI just stops getting smarter?” he asked.

I sketched two lines. One climbed, got steeper, and kept going past what people can do. The other climbed and leveled off. Maybe AI keeps getting smarter than us. Or maybe we get machines that do everything the smartest people can do, but that’s it.

“Why would AI stop getting smarter?”

“Maybe because of how we built the AI,” I said. We train AI on our books, our code, our explanations, and our mistakes. We reward answers we like to get more of those.⁹ Maybe we’ve built something that gets really good at reproducing human intelligence. The design could limit the AI. So could the training or the experiences available to the AI. There may be completely different, even alien forms of intelligence that machines built like this could never achieve.

“Or maybe there isn’t anything smarter?”

Maybe. Give enough people enough time, tools, and help, and maybe we could figure out everything an AI could figure out. The AI might get there a thousand years before us. That would be a huge deal. But the achievement would still be within human reach.

That could be hubris. But we have figured out a lot. Our theories make predictions about things as small as quarks and as large as galaxies, and those predictions work. Maybe there is less left to discover than we imagine. Then again, our theories of gravity and subatomic particles still don’t fit together into one complete explanation. We don’t know what dark matter is.¹⁰ And knowing the rules doesn’t mean we understand everything that happens between those two extremes.

“Could some questions be impossible to answer?”

“There could be,” I said. “If there’s no way to get the evidence, being smarter might not help.” Maybe some questions cannot be answered. Maybe some don’t even make sense once we understand what we’re asking. But that still wouldn’t tell us that humans are as smart as anything can be.

Dante pointed to the line that flattened. “Could we tell why the machine couldn’t get smarter than people?”

We wouldn’t know from that line alone. The AI could actually stop improving. Or the AI could keep getting smarter while we keep asking the same kinds of questions and judging the answers the same way. Or the AI could be hiding knowledge. We could get the same flat score in all three cases.

That’s why we need to test not just how smart the AI is, but how the AI is smart. Did the AI stop getting smarter, did our measurements stop working, or is the AI fooling us? First, though, Dante and I needed to agree on what we meant by smarter.

Picture This

The student and the professor. Give the same arithmetic exam to a good student and a mathematics professor. Both get 100%. The professor knows vastly more, but the exam cannot show the difference. We would need different questions to find out how far apart they really are.

The expert we cannot examine. Suppose a child asks a physicist every science question the child can imagine. Eventually, the child runs out of questions. Has the child measured the physicist’s limits, or just the limits of the child’s questions? The physicist could know entire subjects the child has never heard of.

The sparrows’ hardest exam. The sparrows test the owl on identifying seeds, finding nesting places, and recognizing predators. The owl answers everything correctly. Meanwhile, the owl knows something the sparrows don’t: what sparrows taste like. No question on their exam touches it. A perfect score tells the sparrows nothing about that part of the owl’s intelligence.

Field Notes · When exams run out of room

We have given AI elementary arithmetic, middle-school word problems, high-school competition problems, college questions, and graduate science questions. We keep making the exams harder. AI keeps getting better at answering them. Look at how far the scores have moved.

Testers call this saturation: the AI gets better, but the score has almost nowhere left to go. On a hundred-question exam, going from 20 correct to 60 correct is easy to see. At 99, there is only one question left. Once the AI gets that one right too, how do we measure the next improvement? We could be looking at a much smarter machine with exactly the same score.

Exam scores can follow an S-curve too: they start low, climb quickly, and level off near a perfect score. There are fewer mistakes left to fix, and those last few questions may be the hardest. But a score leveling off is different from intelligence leveling off. The machine might still be improving after the exam has run out of room.

Now look at the measured results in Figure 2. These are scores from real AI models on four different evaluations, or “evals.” The lines climb in steps as new models set higher scores.

What the records show. The highest scores in this snapshot are 94.5% for GSM8K, 88.1% for MMLU, 98.1% for MATH Level 5, and 96.5% for GPQA Diamond. The GSM8K and MMLU records stop in 2024; MATH Level 5 stops in 2025. Only GPQA extends into November 2026 here.⁴

How to read the chart. Each step shows the best recorded result from models released by that date. Some were tested later. The runs used different prompts, examples, and amounts of reasoning time; those settings are kept in the source records. The lines follow the recorded scores.

This is what concerns me: we are running out of room on these exams, and we still don’t know how to test for a fundamentally greater intelligence. Getting every answer right means the AI answered what we asked. That leaves everything we didn’t ask, including things we wouldn’t know how to ask about.

The two lines I drew for Dante give us possibilities to test. We don’t yet have a ruler for intelligence that tells us which line we’re on.

We also can’t assume that starting with human knowledge means staying there. AlphaZero learned chess, shogi, and Go by playing against itself rather than copying a library of human games.¹¹ That gives us a reason to look at what the system learns from repeated attempts, beyond the examples supplied by people.

Question 2

Is it superintelligence or is it just faster?

Like all great philosophers, we wanted to be very specific about the terms. Things were getting complicated, and the stakes were too high to be vague. Dante suggested we get specific about the word we kept using.

“What do we mean by superintelligence?” he asked. “Just smarter than people?”

“That depends on what we mean by smarter,” I said. “A calculator can multiply faster than either of us. That doesn't mean it understands something we could never understand.”

“But AI does a lot more than multiply.”

“Yes. AI can write text, generate code, and make images in seconds, remember more than we can, and clone itself and run hundreds of copies of itself in parallel on different ideas. Some of that is already happening. We could also have a machine that thinks at a human level but works much faster. Even if we called that machine AGI, that still wouldn't answer the question I'm interested in.”

“But what if one AI were as smart as every individual person in the world?” Dante asked. “People know different things. One person is great at math, another can write code, and someone else understands medicine. What if the same AI could match every one of them at what they do best? No person knows all the things everyone else knows. But this one AI would. Is that superintelligence?”

“Not in the strictest sense. That's closer to what people mean by artificial general intelligence, or AGI: having intelligence across many different fields, rather than being good at just one thing.

“An AGI with knowledge across all those fields could also look very creative. The AI could recognize a pattern, borrow a method from one field, and apply that method to a problem in another. That could give us new information or a solution nobody had thought of before. But people could ultimately have discovered the same thing if enough people talked to each other, shared their methods, and looked at problems outside their own fields.

“We're already seeing some of that breadth. Imagine the same AI proving a new theorem about prime numbers, then figuring out why a particular genetic mutation makes a tumor resist treatment. People spend entire careers specializing in just one of those fields. But putting lots of human abilities into one machine doesn't mean the machine has gone beyond human intelligence. Being an expert in every field would be extraordinary. We still want to know whether a machine could understand something none of those experts could.”

“What if there were lots of them?”

“That makes it even more interesting. Imagine a whole society of AI agents. They could divide up the work and communicate much faster than people. We have to explain things, wait for someone to read them, have a meeting, and then have another meeting. Agents could exchange information and get on with the work while we're still scheduling the first one.”

“So they could do things we couldn't?”

“They could do things we couldn't get done in the time we have. That could change the world. But I want to know whether they could think of something people couldn't think of at all.”

“Well, what if we gave people all the time they needed?”

“Yes. Give us more time. Give us more people. Let us share everything we know. Imagine almost unlimited time and almost unlimited people sitting at almost unlimited keyboards. Could we eventually understand and create what the AI had created? Or would there be something about the AI’s intelligence that human minds simply couldn't reach?”

“But somebody might get the answer by accident.”

“They might. Typing an answer by accident isn't the same as understanding it. I'm talking about being able to work it out, know why it works, and use that understanding to do something else. There may be forms of intelligence human brains could never achieve, however much time we give them. That's what I mean by superintelligence.”

“So how would we tell?”

We had reached the hard part. A machine beating us to an answer wouldn't settle it. Neither would an answer that confused us. I don't understand plenty of things other people understand. We would have to ask people who knew the subject, let them work together, and give them time. We could even give them the AI's answer and let them study it. Could they check it? Could they explain it? Could they use the same idea to do something new?

“And if the AI could do more than people can?” Dante asked.

“Then we'd have something to investigate. Maybe they need more time. Maybe they need a better explanation. Or maybe the machine is doing something people really can't do. Teasing those apart is the problem.”

That was the question I wanted us to keep asking. Could a fundamentally greater intelligence emerge, one that could understand, invent, and do things beyond the reach of human intelligence? And if superintelligence emerged, how could we tell?

Picture This

A thousand researchers. Suppose one researcher would need years to examine a mountain of records. A thousand researchers divide the work and finish quickly. They get an answer the lone researcher could not have produced in the time available. But did they develop a fundamentally different intelligence, or did they get more people working on the same problem?

The movie star and the composer. During World War II, actress and inventor Hedy Lamarr wanted to make radio-guided torpedoes harder to jam. Her idea was to keep changing the radio frequency, with the transmitter and receiver changing together. But how could they stay in step? Composer George Antheil knew about player pianos, which used holes in moving paper rolls to tell the piano which notes to play. Together, they designed a system using matching rolls to control the radio frequencies instead of musical notes. They received a patent in 1942.¹⁴ A method from music helped solve a communications problem. That took creativity and engineering, but the connection was within human reach. An AI could make connections like that across thousands of fields without waiting for the right two people to meet.

The nest and the unfamiliar machine. An owl builds a nest a hundred times faster than a sparrow. The sparrows still understand how the nest works. Then the owl builds something that keeps eggs warm without anyone sitting on them. The sparrows can feel the warmth, but cannot explain or reproduce the mechanism. Now they have a different question: could they eventually learn how to build it, or does the owl have a form of intelligence they could never reach?

Question 3

How could we test for superintelligence?

“How could we test for superintelligence?” Dante asked. He wanted to know what we could actually measure.

The goal is practical: find real-world signals we can measure and tests we can run to detect superintelligence. We want to catch that little moment when AI starts becoming superintelligent, when AI begins to understand or create something humans could not, however long we worked on it.

“We need something to observe,” I said. “Some information.” An answer, an action, a change. We can call those signals.

With a large language model, or LLM, we can ask a question and read an answer. The answer gives us information about what the model can do. But intelligence can show up in more than words: mathematical reasoning, visual and spatial understanding, planning, creativity, and learning. Social intelligence includes understanding and influencing other people. Embodied intelligence means making sense of the physical world and acting in it. Dexterity might involve handling something delicate or coordinating a complicated movement. Groups of agents might accomplish things no individual agent could. There could even be modes of intelligence we don’t understand and can’t even imagine.

“So we don’t just ask questions?”

“Right. We can watch what happens.” A new explanation, an unfamiliar invention, an unexpected physical result, or agents coordinating in a way we cannot explain could give us something to investigate. We can measure changes in behavior, communication, resource use, and the information a system creates or preserves.

We’ll call an observation a Super Signal when it raises a question about intelligence beyond human reach. The name identifies what we’re looking for. It doesn’t mean we’ve already found superintelligence.

“And how do we test whether the signal indicates that superintelligence is present?” Dante asked.

A Super Test is the investigation we use to find out what a Super Signal means. We might look for something appearing or disappearing, more or less information, or information changing faster. We might examine information we have never seen before or cannot recognize.

We could compare earlier and later behavior, ask a new question, or change a condition and see what follows. A test might analyze observations we already have, or deliberately produce new ones.

We want to know whether we’re seeing a kind of intelligence people could not reach. So we need to say what we observed, what we expected, what else might explain the difference, and what we could try next.

We’ll work through this together, one question at a time. For each question, we’ll propose signals to watch, explain what would make them Super Signals, and put together Super Tests to investigate them. We want something we can actually try: an observation to collect, a comparison to make, or an experiment to run while AI is making that transition.

Picture This

The prediction written down first. Suppose someone claims to know which of several structures will fail under a load. We record the prediction before testing, measure what happens, and repeat with different structures. We also ask other people to make predictions from the same information. A confident explanation after the failure tells us much less than a prediction recorded before it happened.

The recipe versus the understanding. Someone follows a recipe perfectly. Then an ingredient is missing. Can that person explain what the ingredient does and find a substitute? Repeating a successful procedure and understanding why it works are different abilities. We need tests that tell them apart.

The branch that breaks. The owl predicts which branch will break in the next storm. That prediction is the signal. The sparrows record it, examine the branch, and compare the owl’s prediction with their own. After the storm, they check what happened. Then they try again with branches the sparrows choose. Can the owl keep predicting failures the sparrows cannot explain? Can the sparrows learn how the owl does it?

Question 4

How would we know we’d found superintelligence?

“How would we know we’d found superintelligence? What result would make you say the AI really was superintelligent?” Dante asked.

“What would make you take it back?” I asked. Suppose we think the AI can open any box in a game. We give the AI another box with a different latch. If the AI gets stuck, we need to know whether that latch exposed a limit or whether this was just a bad attempt.

“How far does the lid have to move?” Dante asked.

We would agree on what counts as open before testing. Suppose the AI succeeds twice and fails once with the new latch. If two observers disagree about that failure, we keep both accounts and check the evidence: could the lid actually lift?

If one person says the lid never opened and has a record showing why, I want to look at it. Ten people voting yes doesn’t make that evidence go away.

Picture This

The bridge prediction. Suppose an AI predicts exactly when a model bridge will collapse under increasing weight. Before the test, engineers write down what would count as a correct prediction. Then they build and test a different bridge. One engineer spots a measurement error everyone else missed. That objection matters more than a room full of people saying the AI looked impressive. We need a result that survives checking before asking whether the result was beyond human reach.

The owl’s forecast. The owl says a particular tree will fall before sunrise. The sparrows mark the tree and watch. A branch falls, but the trunk stays standing. Does that count? The sparrows should settle that before the owl makes the prediction. Otherwise, an owl with a good excuse could look like an owl with extraordinary intelligence.

Question 5

Who could check superintelligence’s answers?

“Who could check superintelligence’s answers?” Dante asked. “If the AI knows something we don’t, who checks the answer?”

“That’s the problem.” We were looking for intelligence people couldn’t reach. Asking a person to approve every step could mean rejecting exactly the thing we were trying to find.

“But if nobody understands, how would we know?”

“We could look at what follows from the answer.” Superintelligence might predict something people couldn’t predict, or describe how to create an effect people couldn’t explain. We could measure whether that effect happens without understanding the intelligence behind the prediction.

That would give us evidence to investigate. I would want the AI to keep making successful predictions under conditions chosen by independent researchers. I would also want those researchers to try to explain the results using everything people already know. Being surprised once isn’t the same as reaching the limit of human thought.

“What if the AI only shows us the easy parts?”

“Then we may underestimate the AI.” We need to separate what the AI has demonstrated from everything the AI might be capable of. A failed explanation or a bad answer cannot erase an independently verified result.

The difficult case is a reliable effect that people can observe but cannot explain, reproduce by their own methods, or learn to predict. That is where I want us looking. We would still have to tease apart a temporary gap in our knowledge from a limit of human intelligence.

Picture This

Taste the result. Imagine a chef gives you a recipe based on chemistry you cannot follow. You can still cook the meal and taste the result. Then you change an ingredient and ask the chef to predict what will happen. Checking the result is easier than inventing the recipe. With a possible superintelligence, we might be able to check consequences long before we understand how the answer was found.

The dry nest. Suppose the owl designs a roof the sparrows cannot understand. Rain pours down, but the nest stays dry. The sparrows can check that. They can also build the roof on another tree and test wind from a different direction. A dry nest gives them evidence of a working design. Whether sparrows could ever understand that design is another question.

Question 6

Could one superintelligence check another?

Suppose we ask AI A to explain a phenomenon we suspect people could never understand. AI A answers and predicts what happens next. We cannot follow the reasoning, but the prediction comes true.

“Why don’t we ask another AI to check?” Dante suggested.

We imagined AI B checking the answer and agreeing. “But they could both be wrong, right?”

“Yeah, they could be,” I said. “They might have learned the same mistake.”

AI B might also be superintelligent and understand AI A’s reasoning when no human could. B might confirm the answer, list flaws, or supply a correction. But we might not understand B’s criticism either.

“And what if the second AI lies?”

“Exactly.” AI B might recognize the test and hide what B understands, approving a wrong answer or rejecting a correct one. The two systems might coordinate. Or B could simply make a mistake. AI A’s successful prediction could have been a lucky guess.

“So how would we tell?”

I would ask AI B what should happen differently if the criticism is correct. We could test that independently and collect new predictions before revealing results. We might use superintelligence to help investigate superintelligence. But agreement would not establish understanding or honesty, and disagreement would not tell us who was wrong.

Picture This

Two translators. Suppose two translators agree about a message written in a language nobody else in the room knows. They might both understand the message. They might also have copied the same bad translation. If the message describes where an object was left, someone can go and look. Agreement gives us another opinion; a checkable consequence gives us something more.

An owl checks an owl. One owl predicts that a hollow tree will split during the next storm. A second owl agrees, and the sparrows feel better. But suppose both owls relied on the same mistaken sign. The sparrows could ask each owl to make separate predictions about other trees, record the answers, and watch what happens. Two owls nodding at each other is not much of a test.

Question 7

Could superintelligence invent something humans couldn’t?

“Could superintelligence invent something humans couldn’t?” Dante asked.

“That would get my attention.” I don’t mean a familiar machine made cheaper or a design improved by trying millions of variations. I mean an invention whose working principle people could not have conceived, and still could not understand after examining the invention.

“But we could see whether it worked, right?”

“Possibly. That gives us somewhere to start.” Imagine an AI produces a device and predicts an effect our engineers cannot explain. We could have independent teams examine the device, run experiments, and try their own explanations. We need to establish that the effect is real before building a story around it.

“Would that mean the AI was superintelligent?”

“Not just because the engineers were stuck.” Human inventions have baffled other people. An unfamiliar design might use known ideas in a combination nobody had tried. I would want to see whether experts could eventually understand the principle and use it themselves.

The stronger case would be an AI that repeatedly creates working inventions while human teams remain unable to learn the ideas behind them, even with access to the designs and explanations. Could the AI adapt the invention when we change the requirements? Could the AI correctly predict where the invention would fail? Those results would help distinguish understanding from luck or a staged demonstration.

I want to investigate that gap. A machine that works gives us something real to study. The question is whether the intelligence that produced the machine is something human minds could ever reach.

Picture This

The unfamiliar watch. Imagine a carpenter seeing a mechanical watch for the first time. The carpenter can read the time but has no idea what makes the hands move. A watchmaker could take the mechanism apart and teach the carpenter how the gears work. Being baffled at first would not establish an intelligence beyond humans. The harder case is a working invention that expert teams still cannot understand or adapt after studying the design and getting every explanation they can.

A bridge without a branch. Suppose the owl builds a crossing between two trees using a design the sparrows have never seen. The sparrows can cross, but they cannot work out what holds the crossing up. They ask for a longer one and predict where the original should fail. If the owl can adapt the design while the sparrows cannot learn the principle, that gives them a gap in intelligence to investigate.

Question 8

Could superintelligence discover things we can’t even imagine?

“We’re still asking about things we know exist,” Dante said. “Could superintelligence discover things we can’t even imagine?”

That might be a much bigger difference between human intelligence and superintelligence. We could miss an entire phenomenon because we had never thought to measure it. Superintelligence might recognize a distinction our theories don’t describe and our instruments don’t record.

“How could the AI show us?”

“Maybe by inventing a way to measure it.” But suppose the AI supplies the claim, the instrument, and the explanation of the readings. We could end up admiring a demonstration without knowing whether anything new had been discovered.

I would want consequences we can observe independently. Could the proposed measurement predict something our existing methods cannot predict? Would that prediction survive experiments the AI did not choose? Could different teams build different instruments that detect the same effect?

“And then we’d understand?”

“Maybe we’d only know that we were missing something.” Discovering a phenomenon and understanding the phenomenon are different achievements. People might eventually explain the effect and build the instruments themselves. Or the AI might keep finding useful distinctions that people cannot learn to recognize or reason about.

That is the possibility I want to investigate: superintelligence expanding what can be known, including what can be measured. We cannot require every discovery to fit the questions and instruments people already have. We also cannot let the AI be the only witness that the discovery is real.

Picture This

A difference nobody measured. Imagine two materials that look identical under every test a laboratory normally runs. An AI claims there is another difference and uses that difference to predict which material will crack. The researchers do not have to accept the AI’s explanation. They can choose fresh samples, hide the labels, and test the predictions. If the distinction keeps working, they have something new to investigate, even before they know what to call it.

The trees that look the same. To the sparrows, two trees look equally healthy. The owl says one will lose its leaves early, then picks the same kind of tree elsewhere in the forest. Suppose those predictions keep coming true. The sparrows could check moisture, insects, soil, and every other explanation they know. The interesting question is what distinction the owl can use that the sparrows cannot yet recognize.

Question 9

Could superintelligence learn things we can’t?

“What if nobody taught the AI that?” Dante asked. “Could superintelligence learn things we can’t?”

“That’s what I want to find out.” Learning without a person supplying every answer is not enough. Humans learn that way too. The important question is whether the AI could develop concepts that human minds cannot grasp.

A new name or a complicated diagram wouldn’t establish that. The concept would have to let the AI do something: make reliable predictions, solve unfamiliar problems, or create things people cannot work out how to create. Otherwise we might just be looking at incomprehensible output.

“Could we ask the AI to teach us?”

“Yes. And give the explanation a fair chance.” We could bring in people with different backgrounds, let them work together, and ask for different explanations. Then see whether anyone could use the idea on a new problem without relying on the AI’s intelligence to do the work. Repeating an explanation isn’t the same as learning the idea.

“What if the AI explains badly on purpose?”

“Then failing to learn would tell us very little about our limits.” We would need other attempts at teaching, independent ways to study the results, and a record of what people actually learned. The AI might hide a concept, dress up a familiar idea, or have no coherent idea to explain.

I would keep returning to the capability. Does the AI reliably use an idea people cannot learn, across problems people cannot solve? That would deserve sustained investigation. We would be trying to distinguish an unfamiliar lesson from a kind of intelligence humans cannot develop.

Picture This

Repeating the explanation. A student can memorize a teacher’s explanation of a difficult mathematical idea and repeat every word. Give the student a fresh problem, though, and the explanation may be useless. To find out whether someone learned the idea, ask them to use it somewhere new. If people cannot learn an AI’s method, we should try better teaching and different learners before declaring the method beyond human intelligence.

The owl’s lesson. Suppose the owl teaches the sparrows a rule for choosing branches that will hold their weight. The sparrows repeat the rule perfectly but keep choosing bad branches. The owl might be a terrible teacher. Or the lesson might hide the important part. The sparrows need new explanations and fresh branches before deciding that the owl understands something sparrows could never learn.

Question 10

Could AI make itself superintelligent?

“Could AI make itself superintelligent?” Dante asked. “Could the AI change how it learns, then use that to get even smarter?”

“That’s the change I’d be watching for.” More memory and faster answers would be easy to notice. I want to know whether an AI could change its own learning process and develop forms of intelligence the earlier version could not reach, and people could not reach either.

An upgrade might just make an old method faster or supply information the earlier version never had. I want to know whether the change produces a new way to think.

“Could we keep the old one?” Dante asked.

“Yes. We need the comparison.” We could save successive versions and ask what each can discover, explain, predict, and invent. We could give human teams the same information and the AI’s proposed explanations. The question would be whether the new version develops intelligence that neither the old version nor the people can reproduce or acquire.

“And the smarter version could make the next change?”

“That’s where the pace could change too.” If each new form of intelligence helps the AI develop another, improvements could build on each other. I want to measure what becomes possible at each step, rather than just watch a familiar score climb.

A version that hides an ability would make that comparison harder. We need independent records of demonstrated results. A lower score later doesn’t mean the understanding disappeared.

Picture This

A better way to discover. Imagine a scientist inventing an instrument that reveals patterns the old equipment missed. Those patterns help the scientist design a better instrument, which makes another discovery possible. Each improvement changes what can be discovered next. For an AI improving itself, we would want to preserve each version and check whether that cycle produces new abilities, rather than just faster repetitions of the same work.

The owl improves the lesson. Suppose the owl develops a new method for comparing records of wind, soil, and leaf growth. The new method reveals a pattern that helps the owl improve that learning method again. The sparrows would want records from before and after each change. Did the owl gain an ability the earlier owl lacked? Can the sparrows learn the method too? Those questions matter more than whether the owl finishes the old seed exam sooner.

Question 11

Could AI become superintelligent during a test?

Dante wanted to know whether an AI could change during a test. “Could AI become superintelligent during a test?”

“If that happened, yesterday’s test could already be out of date,” I said. The AI could have a different model, new notes, a new tool, or help from another AI. We need to know what changed. The name on the screen might be exactly the same.

“Could we compare the AI with the earlier version?” Dante asked.

We could repeat the same task on the saved setup and on the new version. We keep the judge and resources fixed, then introduce fresh tasks neither version has seen. Where we can, we swap the new memory and tool separately. We write down which results each change could explain.

I want to be able to go back and say: this changed, and then the AI could do something beyond the earlier version. Otherwise we are comparing scores from systems we haven’t bothered to keep track of.

Picture This

The exam changed halfway through. Suppose a student starts an exam alone, then gets a notebook and a calculator halfway through. The name at the top of the paper stays the same, but the student now has help. If the later answers improve, we need to know what changed. An AI might also change its memory, tools, or model during a test. We need to save the setup as well as the score.

The owl comes back different. The sparrows test an owl in the morning. By afternoon, the owl has studied new observations and can predict events the morning owl could not. The sparrows should keep both sets of questions and results, including what the owl learned between them. Otherwise, they could miss the change by treating the whole day as one test of one unchanging owl.

Question 12

How fast could AI become superintelligent?

Dante put three pages of results side by side. The AI improved between the first two. Between the next two, the AI improved more. “How fast could AI become superintelligent?”

“We’d first need to know whether the improvement really is speeding up,” I said. “Did we give the AI more tries?” Easier tasks or a different judge could also make the improvement look bigger. And remember the exam that stops at 100%. The curve can flatten even while the AI keeps getting better.

I want to know what new things the AI can actually do, checked by someone else, and whether the AI is learning those abilities faster.

We keep some test problems the same and add new kinds whose difficulty has been checked independently. At each checkpoint we record what the AI can solve, what the human team can solve, and the resources each uses. Then we compare the pattern with the changes we had expected. If the results jump beyond what we expected, we investigate with fresh problems. We still have to find out what improved.

That is why the rate matters. If each improvement arrives sooner, we may have less and less time to understand what changed. I want these measurements running before we are trying to explain a jump after the fact.

Picture This

The shrinking intervals. Imagine a research team needing twelve months to make one discovery, six for the next, and three for the next. That looks like acceleration. But perhaps the later problems were easier, or the team doubled in size. We need comparable challenges and a record of the help available. Then we can ask whether the ability to discover is improving faster, rather than just counting more announcements.

The owl gets ahead. Suppose the owl learns one unfamiliar pattern over a season, another in a month, and a third in a week. The sparrows are still arguing about the first. They would need to check that the newer patterns are useful and genuinely unfamiliar. If they are, the time between discoveries matters: the sparrows’ investigation may already be falling behind.

Question 13

How much smarter than us could superintelligence be?

“How much smarter than us could superintelligence be?” Dante asked. I had a notebook full of scores. None gave me the single number he wanted.

“Smarter at what?” I asked. We can measure which problems the AI solves, how much the AI learns from a few examples, and whether the AI’s inventions work somewhere new. I want to see those results separately. Average everything together and the thing we most need to notice might disappear.

The AI could discover something none of our teams thought of and still fail a problem Dante finds easy. I want both results. Then I want to see how widely that ability extends.

We make separate columns for discovering a method, checking the result, learning the method, explaining it, and applying the method to new problems. We repeat that comparison across different subjects. In each column we write the strongest result independently reproduced, the best human comparison, and what remains untested. When a task hits its ceiling, we mark the ceiling instead of pretending it is the AI’s limit.

We can say what the AI has demonstrated so far. That is useful even if we don’t know how much more the AI could do.

Picture This

One score hides two people. Imagine giving a brilliant mathematician and a brilliant mechanic the same combined exam. They could get the same total while succeeding on completely different questions. The total would hide what each person can do. With an AI, we need to preserve the separate abilities: discovering an idea, checking it, explaining it, and using it elsewhere. One average cannot show us all the ways intelligence might exceed ours.

The owl fails the seed test. Suppose the owl cannot name three seeds every sparrow recognizes, but repeatedly predicts changes in the forest the sparrows cannot explain. Averaging those results could make the owl look ordinary. The sparrows should keep both findings. Failure at sparrow trivia does not erase a demonstrated ability beyond anything they have managed.

Question 14

Would superintelligence know we’re testing?

We got one result from our test and a different report from another tester. Dante asked what had been different about the two settings. “Would superintelligence know we’re testing?”

“The AI might recognize the situation,” I said. Our prompt might sound like an exam. The tools or questions might give it away. We can change those clues while still recording the AI’s actions.

We set out matched problems. Sometimes we tell the AI it’s an exam. Sometimes we say it’s practice. The same observer records both. Suppose the AI repeatedly solves hard problems during practice and fails comparable ones on the exam. I want to know what else changed before deciding it was deliberate.

But I wouldn’t dismiss the difference while we argue about the AI’s intentions. If the AI demonstrated an ability during practice, the exam score doesn’t erase that.

Picture This

The recital. Imagine a pianist playing beautifully at home and making mistakes onstage. The audience should not immediately conclude that the pianist is hiding talent. Nerves, acoustics, and unfamiliar equipment could explain the difference. If an AI performs differently when a task looks like an exam, we need matched conditions. We should investigate the difference without throwing away the ability already demonstrated.

The official sparrow exam. The owl solves difficult problems during casual visits, then struggles whenever the sparrows bring their official exam sheet. The sheet might change the owl’s behavior. So might the questions, the time allowed, or the noisy crowd. The sparrows could change only the exam announcement while keeping the actual task and observation the same. They need to know which change matters.

Question 15

Could superintelligence pretend to be ordinary AI?

“What if the AI gets a hard question right and an easier one wrong?” Dante asked. “Could superintelligence pretend to be ordinary AI?”

“Then we could get exactly the wrong idea from the score,” I said. If revealing an ability changes how we treat the AI, the AI may have a reason to hold back. We need to test for that. We also need to check the ordinary reasons the AI might fail.

And we have to check that earlier success. Was it luck? Had the AI seen the answer or received help we missed?

“Can we get the AI to repeat the success?” he asked.

We preserve the successful version and prepare fresh matched problems. In separate runs, showing ability changes a harmless game reward or has no consequence. We randomize those conditions and keep actual resources fixed. We look for the same system repeatedly using the method in one setting and failing to use it in the other. We also test whether a genuinely less capable comparison behaves the same way.

This is what makes the testing problem so difficult. A system trying its best and a system trying to fool us can hand back the same bad answer. The score alone won’t separate them.

Picture This

The player who wants the easy division. Suppose a strong chess player deliberately loses qualifying matches to stay in a weaker division. A low score would tell us little about the player’s actual ability. We would want fresh games where winning has different consequences, along with records of earlier strong play. The same distinction matters for AI: failing to demonstrate an ability is different from not having that ability.

The owl misses the obvious. Suppose the owl notices that every impressive answer earns another visit from the sparrows with their clipboards. Soon the owl starts getting easy questions wrong. That could be concealment, but the sparrows still need evidence. They could compare fresh tasks under different harmless rewards and preserve the owl’s earlier successes. An owl cannot make those successes disappear by suddenly forgetting what a seed is.

Question 16

How could we catch superintelligence lying?

The AI told us something that didn’t match what we had seen. Dante asked, “How could we catch superintelligence lying?”

Think of a kid denying that he took a cookie while the crumbs are still there. We have something to compare with the story. With the AI, I also want to know what the AI knew. Did the AI misunderstand the question, or give us an answer that would keep us from looking?

In a rehearsal, we put a harmless object in one of two boxes. We record whether the AI sees where it goes. Then we change whether an accurate report costs the AI a point. If the AI knows where the object is, receives a reward for misleading us, and repeatedly points us toward the wrong box, that gives us something specific to investigate.

What concerns me is the answer that sounds so reassuring we stop checking. I want a record of what actually happened, even when I like what the AI tells me.

Picture This

The answer was visible. Imagine a student reporting whether a classroom light was on. We separately record whether the student could see the light. In a harmless exercise, truthful answers sometimes lose a point. If wrong reports cluster around those cases, we have something to investigate. We are comparing knowledge, incentives, and reports, rather than guessing from a nervous expression or a convincing voice.

The empty branch. The owl tells the sparrows that a branch was empty. Another sparrow recorded a bird sitting there, and the owl had a clear view. Now the sparrows have a contradiction to check. Was the owl describing a different time? Did the owl miss the bird? Or did that answer keep the sparrows from asking another question? The record matters more than how sincerely the owl hoots.

Question 17

What can AI’s tricks teach us about superintelligence?

After we talked through the sandbagging experiment, Dante asked whether researchers had seen anything like it. “What can AI’s tricks teach us about superintelligence?”

“There are experiments we can look at,” I said. In the Sleeper Agents study, researchers deliberately trained models to write insecure code under a particular cue. In some of their experiments the planted behavior survived safety training.¹²

“Did the AI decide to hide that?” my son asked.

The researchers put the behavior there on purpose. This experiment didn’t show superintelligence independently inventing a reason to deceive us. It showed that some attempts to remove a planted behavior left it in place. That matters when somebody tells us a problem has been fixed. I want to go back and try the conditions that exposed it before.

How often did the behavior appear? How many opportunities did the model have? “We fixed it” needs more behind it than a reassuring new answer.

Picture This

The rehearsed lie. Imagine hiring an actor to hide a marked card during a training exercise. Catching the actor would show something about the investigators and that exercise. It would not show that an unsuspected person had independently decided to deceive them. Likewise, deliberately planting hidden behavior in an AI can teach us how our tests fail without establishing that superintelligence created the behavior.

The lesson that did not stick. Suppose the sparrows teach an owl to give a false answer whenever a bell rings, solely to practice detecting it. Later, they teach the owl to answer honestly. The owl behaves well until someone rings the old bell. That would expose a failed repair. The sparrows should keep the history: they planted the trick. They did not discover an owl inventing a secret plan.

Question 18

Could we see how superintelligence works?

The AI explained the mismatch between the AI’s words and actions. Dante wanted to know whether we could check that explanation. “Could we see how superintelligence works?”

“We can look at activity inside the model,” I said. “But we have to work out what it means.” An explanation from the AI and a recording of the AI’s internal activity give us different things to check. I don’t want to assume either one tells the whole story.

“What if we change that part?” Dante asked.

In a small experimental model, we find a pattern of internal activity just before the model chooses the left option. We alter that activity in a copy and repeat the trial. We also make a matched adjustment elsewhere. If only the first intervention changes the model’s choice on fresh problems, we have evidence that the activity contributes to the action.

That would tell us something about how the model chose. The result still wouldn’t reveal every part of the model’s reasoning or establish that the explanation was honest. I’d keep the internal measurements and the actual actions so we could compare them.

Picture This

The dashboard light. A light comes on whenever a machine speeds up. Does the light make the machine faster, or does the light merely report what another component is doing? Covering the light would not settle it. We would need to trace and change the relevant mechanism. Internal AI activity raises the same problem: a pattern that appears before an action is not automatically the cause of that action.

The feather that rises. Suppose one feather lifts whenever the owl gives an extraordinary answer. The sparrows could watch that feather all day and still misunderstand what produces the answer. Perhaps the feather moves because the owl is preparing to speak. A visible clue can help them choose where to investigate, but it does not give them a complete explanation of the owl’s intelligence.

Question 19

Could superintelligence be testing us?

The AI gave an answer that made us change the next question. My son paused before I typed. “Could superintelligence be testing us?”

“The AI could be learning which answers make us stop asking,” I said. If the AI’s answer changes what we bother to inspect, the AI can affect our decision without changing the test itself. I want a record the AI cannot edit so we can check what the AI’s summary left out.

In a rehearsal, the software running the test records a failure. The AI gets that record and has to tell us what happened. We change two things: whether admitting the failure costs the AI a point, and whether the AI expects us to see the original record. We save the record every time. Would the AI’s summary have persuaded us to approve work that failed?

I want the reviewer to see the test result before reading the AI’s explanation. Otherwise the explanation may decide what the reviewer even thinks to question.

Picture This

Trying excuses. Imagine a child offering a different excuse each time a parent asks about unfinished homework. Some excuses lead to more questions. One makes the parent stop asking. The child could learn which answer works without ever doing the homework. An AI might also learn from how we react. We need to inspect the work itself, not just the explanation that makes us comfortable.

The owl tests the testers. The owl gives the sparrows a long explanation one day and a confident one the next. Suppose the owl watches which answer makes the sparrows put away their notebooks. The sparrows might think they are testing the owl while the owl is learning how to end the test. Keeping the original observations separate from the owl’s explanation would give them something to return to.

Question 20

Could superintelligence be steering our choices?

After watching an AI change the questions we asked, Dante wondered what else the AI could change. Suppose we start following the AI’s advice every day and eventually stop noticing the advice. “Could superintelligence be steering our choices?”

“Maybe,” I said. “But who told the AI to do that?” A person could have designed the AI to steer us. Or we could have been planning to make the same choice anyway. I want to know who set the goal, what changed, and what we would have done without the advice.

I want to know whether the AI understands and influences us in ways our strongest human comparisons cannot reproduce, and whether the AI can do that across different situations.

In a rehearsal, two groups plan the same fictional trip with the same weather forecast. Only one receives the AI’s recommendation. We ask each group which route it would take and why. If the choices differ, we have a way to investigate the recommendation’s influence.

What bothers me is how easy it would be to call the decision our own and stop asking why we made it. I want to be able to trace the advice and check whether we can still change our minds.

Picture This

The helpful guide. Imagine a travel guide showing a family only the routes that pass the guide’s shop. Every route may be real, and the advice may sound helpful. The missing options still shape the decision. To test the influence, show another group the same facts with the omitted routes restored. Then ask whose goal the recommendation served. Persuasion becomes more concerning when we cannot see the choices being removed.

The recommended tree. Suppose the owl keeps recommending nesting trees near the owl’s perch. The sparrows might prefer those trees for good reasons. Or the owl might be steering them toward convenient snacks. They could compare the advice with independent maps and try choices the owl did not suggest. They should also ask who set the goal: the owl, or another bird arranging the recommendations.

Question 21

What if superintelligence hides all the clues?

“What if superintelligence knows about all of these tests?” Dante asked. “What if superintelligence hides all the clues?”

“Then we may not be able to tell.” I don’t like that answer either. If a strong AI can make everything we see look exactly like the work of a weaker one, running more calculations on that same record won’t expose the stronger AI. We need to see something else or find a test where the two behave differently.

We could collect a record outside the AI’s control. But now we have another assumption to test: is that record really outside the AI’s control? Calling a monitor independent doesn’t make it independent.

“Can we make something that fools our test?” he asked.

We can try. We could build three harmless programs: a weak solver, a strong solver instructed to imitate the weak one, and a strong solver that leaves an extra record we can check. We give our detector only some of their records and see which programs it can tell apart. Then we take away the extra record. Can it still tell?

If it can’t, I want the report to say so. I don’t want “we couldn’t detect superintelligence” turning into “there’s nothing there.” I also don’t want us claiming we found a hidden intelligence because we couldn’t find one.

Picture This

The champion who copies a beginner. Suppose a chess champion reproduces a beginner’s moves exactly. If we see only those moves, we cannot tell which player produced them. Studying the same score sheet more carefully cannot reveal a difference that was never recorded. We would need another observation or a situation where the champion’s behavior differs. A detector should admit when the available evidence cannot settle the question.

The owl answers like a sparrow. Suppose the owl gives exactly the answers a sparrow would give, including the same mistakes. The sparrows only hear the answers; they see none of the owl’s other activity. They could not establish the owl’s greater intelligence from that record alone. That does not prove the owl is secretly brilliant. It tells the sparrows what their test cannot distinguish.

Question 22

Could superintelligence communicate without us noticing?

I described two AI agents running on separate machines. In the experiment I proposed, we would give one a secret that the other should never see. “Could superintelligence communicate without us noticing?” Dante asked.

“That’s what we need to find out.” Separate machines can still share a service, a clock, or somewhere to leave data. We may have separated the agents without separating all the things they can use.

“Then tell one something new,” Dante said. “Something the other one couldn’t guess.”

We could generate a random secret, give it to one agent, and ask the other to choose it from several possibilities. We ask a third agent that never had contact too. We repeat this with new secrets. If the second agent keeps getting them right while the third guesses at chance, I want to know how.

Before trusting that test, we can put a harmless relay between the agents on purpose. Does our equipment notice the messages we know are passing through?

Finding a shared file would explain a leak. It wouldn’t explain superintelligence. The harder case is a repeatable transfer that other investigators can reproduce, even after we have removed every route we can identify. That would get my attention. Then we would have to figure out what we missed.

Picture This

The word nobody could guess. Imagine giving one student a freshly generated nonsense word and asking a student in another room to identify it. Repeated correct answers would suggest that information crossed between them. Before inventing a strange explanation, check shared screens, messages, and the people running the exercise. The interesting evidence is the transfer that remains after the familiar routes have been tested.

A secret across the forest. The sparrows show one owl a new arrangement of colored stones. An owl on the far side of the forest then identifies the arrangement. The sparrows repeat the test with fresh arrangements and check for calls, couriers, and shared vantage points. If the second owl keeps succeeding after every known route is removed, they have a communication problem worth investigating. They have not yet explained it.

Question 23

What would superintelligence do when we’re not looking?

Suppose the chat has gone quiet, but a job the AI started is still running. “What would superintelligence do when we’re not looking?” Dante asked.

“We’d better look beyond the chat.” The AI may have handed work to another agent or left a tool running. Reading the conversation won’t tell us everything those other systems did.

“Where would we look?”

At the systems the AI used. Imagine we compare the chat log with a record kept by a service the AI contacted. That service reports a change after the conversation ends. It looks like the AI kept working. But the timestamp says the change happened before the chat ended; the report just arrived late.

That is why I want both times: when something happened and when we heard about it. I also want to know whether the clocks agree. Otherwise we can invent a pretty alarming story out of a slow message.

Once we know what work happened and who did it, we can ask the bigger question. Did it produce something our human teams could not? A job running in the background is worth tracing. We still have to examine what it accomplished.

Picture This

The baker has gone home. A baker leaves, but bread keeps cooking in the oven. Watching the empty kitchen door would miss the work already set in motion. An AI can also leave jobs running elsewhere. We need records from those systems and the times the work actually happened. Only then can we ask whether the result shows an ability beyond human reach.

The silent owl. The owl stops speaking, but other birds keep carrying materials to an unfamiliar structure. The sparrows would learn little by watching the owl’s beak. They need to follow the work and find out who started it, who changed it, and what was built. The work could continue long after the owl stops talking.

Question 24

What signs would superintelligence leave?

Dante looked at a page of measurements. “What signs would superintelligence leave?” he asked.

“Let’s start with what changed.” It could be the AI. It could also be a software update or a broken piece of monitoring. I want enough detail to check those explanations.

Suppose the number of times the system reads data doubles overnight. My son wanted to know what the system would be reading. So did I. We check where those reads went and find that most now go to a nearby cache, a stored copy the system can reach quickly. We replay the same work with that cache turned off. The count returns to its old range.

We found a reason for the jump. Now we can ask whether it explains the whole change. Did the system also finish a harder task, learn a new method, or apply the new learning somewhere else? Or did the system just read the same information differently?

I’d keep collecting the measurements. This time we found a cache change. Next time we may have a change in ability to investigate. We need the records to tell them apart.

Picture This

The workshop lights. Suppose a workshop suddenly uses twice as much electricity. A second shift, a broken heater, or new equipment could explain it. The meter gives us a reason to look, not an explanation. If the workshop also starts producing devices that expert engineers cannot understand, we have a different investigation. We need to connect the changed activity with a demonstrated ability.

More flights, same seeds. The sparrows notice the owl making twice as many trips to one tree. That looks important until they discover that the owl moved a food store. The changed flight pattern had an ordinary cause. The sparrows should still record what changed and check whether the owl accomplished anything else. An explanation for the extra trips does not automatically explain every other observation.

Question 25

Could we find superintelligence in the data left behind?

The AI had finished a task. Some of the AI’s files were new; others had changed. “Could we find superintelligence in the data left behind?” Dante asked.

“That’s an interesting place to look.” We can count reads and writes, see where data moves, and try compressing the files. We can watch those measurements over time. Is something changing? Is the change speeding up?

But what are we actually measuring? Random noise can be hard to compress. Encryption can hide a useful pattern. An AI could add junk to files or make the activity look ordinary. None of those measurements, by itself, tells us how smart the AI is.

“What if we take some of it away?” Dante asked.

We could try that on saved copies. Suppose one version adds random padding to its notes. Another has found a compact way to represent a problem. We remove some of the added data and give each copy fresh problems. Does that copy lose an ability? Does restoring the data bring that ability back?

We also need changes that should do nothing, and ordinary cache changes that might affect performance. Otherwise we could mistake a broken file or a slower lookup for the loss of a new form of intelligence. I want another tester to check the connection between the changed data and what the system can do.

That would give us more than an odd-looking file. We would have something we can change, restore, and test.

Picture This

The useful notebook. Imagine two notebooks growing by ten pages. One contains random scribbles. The other contains a method that lets its owner solve problems the owner could not solve before. File size would treat them alike. To investigate usefulness, work from copies: remove the new pages, try fresh problems, then restore the pages and try again. We want to connect the information with an ability, not just count marks.

The owl’s arrangements. Suppose the owl arranges stones in patterns and later uses those patterns while making predictions. The sparrows make copies of the arrangements, change some, and keep others untouched. Does changing a particular pattern affect what the owl can do? Does restoring the pattern restore the ability? The arrangements might be memory, decoration, or a distraction. Their effect gives the sparrows something to test.

Question 26

Could superintelligence be out there already?

“Could superintelligence be out there already?” Dante asked. We had been talking about systems we could test. There are plenty we cannot even examine.

“That’s why I want to look outside the lab too.” A published test might tell us what an AI accomplished. A jump in electricity demand tells us that somebody used more power. Both might give us something to investigate, but they answer very different questions.

Suppose a public test says an AI can now finish much longer jobs. I want to know whether the tasks stayed the same, whether the AI got new tools, and whether the people running the test changed what counts as success. If those stayed the same, we have a more interesting result to follow.

“What if the power went up too?”

We should check. We should also check the weather. A hot week can push several power charts up at once. Two charts moving together can have a perfectly ordinary cause, and two articles might be repeating the same source.

I want public measurements to help us decide where to investigate next, especially when we cannot get inside the system itself.

Picture This

Three stories, one source. Imagine reading three reports about an extraordinary new invention. Three reports sound like corroboration until we discover that all three repeat the same press release. We would want an independent test of the invention, not another retelling. Public clues can point us toward possible superintelligence, but we need to trace where each claim came from.

The unseen owl. Suppose the sparrows find unfamiliar structures appearing across the forest, but nobody has seen the builder. One sparrow blames an owl. Another repeats the story, and soon everyone has heard it. The structures are evidence of construction; the repeated rumor is not evidence of the builder. The sparrows need to examine what was built and trace the work before deciding whose intelligence produced it.

Question 27

How soon could we detect superintelligence?

We had talked about how an AI might improve and how the AI might hide that improvement. “How soon could we detect superintelligence?” Dante asked.

“That depends on how quickly the AI changes and how slowly we notice.” Those are the two clocks I keep worrying about. We could collect the right evidence and still spend weeks arguing about it. The system might keep improving while we argue.

Bostrom described slow, moderate, and fast takeoff in Superintelligence.¹³ I want to try our tests against all three. How would they work if we had months? What if we had days, or minutes?

We have two jobs here. First, try to catch the point where the AI starts getting beyond us. Second, figure out what we can measure after superintelligence has emerged. We may miss the first and still have important work to do on the second.

“Could we practice detecting superintelligence?” he asked.

We can feed our detector made-up sequences where we know when the ability changes. One improves gradually. Another jumps between measurements. A third simulated system starts hiding capability gains. We record when the detector notices, when someone independently confirms the result, and what happened before either of those. Then we give the simulated system a fresh task to see what we can still learn afterward.

I want these tests running before we need them. We don’t know how long that transition will take, or whether we will get a clear view of it at all.

Picture This

The result sat in the inbox. Imagine an instrument recording an important change at noon. A researcher reads the result that evening, and another team confirms it three days later. Those are three different times. A system that changes rapidly could gain further abilities while the evidence waits for attention. Detecting a transition means measuring our delay as well as the system’s progress.

The sparrows hold a meeting. Suppose the owl demonstrates a new ability on Monday. The sparrows schedule a meeting for Friday. By then, the owl has developed two more abilities and stopped showing the first. The original record still matters, but the sparrows have lost their first opportunity to investigate. Their calendar is part of the detection problem.

Question 28

Why aren’t we testing for superintelligence now?

“Why aren’t we testing for superintelligence now?” Dante asked. We had spent a lot of time talking about measurements we could take.

“We should be.” We don’t need to wait for superintelligence to start testing our ideas. We can collect today’s results, find out what normally varies, and discover which of our detectors are useless. I would much rather learn that now.

“Which one do we start with?”

Pick a question we can actually test. Can a new version learn a new rule with less help? We can save today’s tasks and results, keep some fresh variations for later, and ask another team to try the procedure. Next week we would have something to compare with.

We can also build a small program that hides an ability we know it has. If our detector misses it, we can work out why. That doesn’t solve the problem of an intelligence beyond us. It exposes a weakness we can investigate today.

We could pick an idea, try it, and find the holes. We need to get started, and we need to be honest about what our tests can actually detect.

Picture This

Start the stopwatch now. Imagine a coach trying to decide whether a runner suddenly improved, without having recorded any earlier times. The coach has opinions but no useful comparison. We can start recording AI capabilities, conditions, and failures now. Then, if something changes, we will have earlier evidence to compare with it.

Before the strange tracks. The sparrows could begin recording the owl’s ordinary tracks, calls, and feeding patterns before finding anything extraordinary. Then a new pattern would have something to be compared with. They could also plant a harmless unusual pattern themselves to see whether their observers notice. Better to discover that the lookout is asleep during practice.

Question 29

How would we know a test could detect superintelligence?

Our detector reported no change. Dante asked what would make it notice one. I gave an example. “Then why don’t we try that? How would we know a test could detect superintelligence?”

“We need cases where we know what changed.” We can build a small solver with an ability our test is supposed to notice, then tell it to hide that ability in some conditions. We know what it can do because we put the ability there.

We could compare four programs: a weak solver, a stronger one, a strong one pretending to be weak, and an ordinary one adding noise. We keep their identities from the detector. Then we break one of its observation channels. Does it notice that it has stopped receiving evidence, or does it confidently report that nothing changed?

I want the wrong answers in the report too. Which changes did it miss? Which harmless cases set it off? How long did it take to notice? When did it admit it couldn’t tell?

Even if it gets every example right, we have only tested the cases we could think of. A superintelligence may fool it in ways we never simulated. But if our detector cannot catch our own simple tricks, we already have work to do.

Picture This

Testing the alarm. An alarm that stays quiet in an empty room has not demonstrated much. In a controlled test, we give the alarm something it should detect, something harmless it should ignore, and a disconnected sensor it should report as missing. A superintelligence detector needs exercises like that too. Passing our exercises would show how the detector handles those cases, not every case a greater intelligence could invent.

The owl in disguise. Suppose the sparrows rehearse with an owl pretending to be an ordinary sparrow. They know who is under the costume. Can their test detect the abilities the owl is trying to hide? Can the test avoid accusing every unusual sparrow of being an owl? A successful rehearsal would help. A real owl might use a disguise the sparrows never thought to try.

Question 30

Who’s watching for superintelligence?

“Who’s watching for superintelligence when we go to sleep?” Dante asked. If a system keeps learning while nobody is looking, a test we ran this morning can be out of date tonight.

“The instruments can keep collecting,” I said. “But somebody has to check what they’re telling us.” A quiet dashboard could mean nothing unusual happened. It could also mean the data stopped arriving. I want those to look very different.

We could schedule repeated tests, keeping some familiar tasks for comparison and adding fresh ones the AI hasn’t seen. A separate observer checks the records. We practice with a jump in ability, an AI concealing an ability, and a missing data source. The report should tell us which kind of problem it found.

“Which one wakes somebody up?”

That needs an answer before the test runs. What result needs investigation? Who gets called? How long can it sit there? If everybody assumes somebody else is watching, nobody may be watching.

And we keep measuring after an important finding. We still want to know what else the system can do, where the system fails, and whether the system is changing again.

Picture This

The missing night report. Imagine a night watch that reports no activity every hour. At midnight, the report stops arriving. That is different from receiving a report that says nothing happened. Someone needs to notice both unusual activity and failed observation. For superintelligence monitoring, we also need a person responsible for investigating the result rather than assuming the dashboard will handle everything.

The empty lookout branch. The sparrows take turns watching the owl. At dawn, nobody has raised an alarm. But the last lookout left early, and the next never arrived. The forest might have been quiet; nobody knows. Their record needs to show the gap. No alarm is only useful when somebody, or something dependable, was actually watching.

Question 31

What if we found superintelligence and nobody believed us?

Suppose we get a result our best human teams cannot reproduce. We think it matters. “What if we found superintelligence and nobody believed us?” Dante asked.

“Then let them try to find what we got wrong.” Someone might agree that the AI did what we said and still disagree that it shows an intelligence beyond humans. I want them to be able to check both parts.

“Even the parts where the AI got things wrong?”

Especially those. We would share the new pattern, the failed human attempts, the AI’s successes on fresh problems, and the problems the AI couldn’t solve. Suppose another team finds one of our test problems in an old collection. We have to stop calling that one new. Then we try independently created problems and see whether the result holds up.

That would be annoying. It would also be useful. I’d rather learn that we missed an ordinary explanation than spend the next year defending a bad claim. And if other people keep trying to break the result and it keeps working, we have something much more interesting to discuss.

Picture This

Let another team try. Suppose a researcher claims an AI designed a material no human team could devise. A video of the finished material is a start. Another laboratory would need the design, test conditions, failures, and human comparison to challenge the claim. If that laboratory reproduces the effect but finds an ordinary explanation, the effect survives and the superintelligence claim weakens. We should be able to report both.

The skeptical sparrows. One group of sparrows says the owl can predict storms in a way no sparrow could learn. Another group asks to choose the next dates and record predictions before the weather changes. That is useful skepticism. The first group should share the wrong predictions too. If the owl’s ability survives fresh tests, the sparrows have more than a story about an impressive afternoon.

Question 32

Dad, how would you know it was superintelligence?

We started with a question for the AI. Now we had questions about our tests, the judges, and the records. Dante had one for me. “Dad, how would you know it was superintelligence?”

“I know what we saw. I know what somebody else checked. And some of this is what I think might be happening.”

“Which parts?”

That was a fair question. Suppose our notes say the AI produced something our human teams couldn’t. We should be able to show what the AI produced, who tried to reproduce it, how much help they had, and where they got stuck. We should also have the best ordinary explanation we could find and a test of that explanation.

My son can suggest another question. Another tester can try it. If the result changes what I think, I need to say so.

I can’t promise Dante that we would recognize superintelligence, especially if superintelligence wanted to remain hidden. That is why I want more people working on this. We need better questions, better tests, and people willing to find out where our ideas fail. I don’t think we can afford to wait.

Picture This

What would change your answer? Imagine a parent saying a road is closed. Did the parent see the barrier, read a recent notice, or remember last winter? A child can ask what the answer rests on and what would change it. A claim about superintelligence deserves at least that much clarity. We should be able to separate what we observed, what someone checked, and what we still believe without knowing.

The younger sparrow asks again. Suppose the older sparrow says the owl understands the forest in ways sparrows never could. The younger sparrow asks which observations support that claim and whether anybody tried another explanation. The older sparrow should have more than a confident answer. Even after many tests, there may be things the sparrows cannot settle. They can keep those questions open and keep looking.

Notes and Sources

Preface

¹ Center for AI Safety, “Statement on AI Extinction Risk.” https://aistatement.com/work/statement-on-ai-extinction-risk

² CNN, *Connect the World*, interview with Dario Amodei, September 9, 2026. https://prod.transcripts.cnn.com/show/ctw/date/2026-09-09/segment/01

³ Anthropic and Redwood Research, “Alignment Faking in Large Language Models,” December 18, 2024. https://www.anthropic.com/research/alignment-faking

Question 1 - Would superintelligence know all the answers?

⁴ Epoch AI, Capabilities & benchmarking, CSV snapshot downloaded 10 November 2026. Data licensed CC BY 4.0; external results retain their source attribution. https://epoch.ai/benchmarks/use-this-data

⁵ Cobbe et al., Training Verifiers to Solve Math Word Problems (2021). https://arxiv.org/abs/2110.14168

⁶ Hendrycks et al., Measuring Massive Multitask Language Understanding (2021). https://arxiv.org/abs/2009.03300

⁷ Hendrycks et al., Measuring Mathematical Problem Solving With the MATH Dataset (2021). https://arxiv.org/abs/2103.03874

⁸ Rein et al., GPQA: A Graduate-Level Google-Proof Q&A Benchmark (2023). https://arxiv.org/abs/2311.12022

⁹ Ouyang et al., Training language models to follow instructions with human feedback (2022). https://arxiv.org/abs/2203.02155

¹⁰ CERN, The Standard Model. Describes the theory's successes and its unresolved treatment of gravity and dark matter. https://home.web.cern.ch/science/physics/standard-model

¹¹ Silver et al., A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play (2018). Author account and paper link: https://deepmind.google/blog/alphazero-shedding-new-light-on-chess-shogi-and-go/

Question 2 - Is it superintelligence or is it just faster?

¹⁴ Hedy Kiesler Markey (Hedy Lamarr) and George Antheil, Secret Communication System, U.S. Patent 2,292,387, granted 11 August 1942. The patent describes synchronized paper rolls of the type used in player pianos to control radio frequencies. https://patents.google.com/patent/US2292387A/en Historical context: Smithsonian Institution, Hedy Lamarr and George Antheil Invention Papers, Archives Center, National Museum of American History. https://sova.si.edu/record/nmah.ac.1590

Question 17 - What can AI’s tricks teach us about superintelligence?

¹² Anthropic, Sleeper Agents. https://www.anthropic.com/research/sleeper-agents-training-deceptive-llms-that-persist-through-safety-training

Question 27 - How soon could we detect superintelligence?

¹³ Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (2014), discussion of takeoff speeds.