XXI. Scaling Error

You create knowledge. And the process is fallible.

You can create a bad explanation, preserve it, embody it in objective knowledge, build institutions around it, encode it into software, and use tools to amplify it. Nothing in the epistemology places a ceiling on the resulting error.

A mistake can remain private; it can also become catastrophic.

That does not defeat optimism. It shows why optimism cannot become complacent. And why even the most catastrophic risks demand good explanations.

1. Catastrophic Error

Fallibility is easy to accept when the consequences are small. You misremember a name. A scientist chooses a bad model. An engineer makes a design error. A government adopts a failed policy. Criticism arrives, and the error is corrected.

Nothing in fallibilism, however, says that reality must leave enough time for correction. A bridge can collapse. A war can begin. A dangerous organism can escape. A technological system can propagate an error faster than anyone can understand it. A civilization can act on an explanation whose failure destroys part of the machinery needed to criticize it.

Abstract it.

The structure is simple:

$$\text{fallible explanation} \rightarrow \text{objective embodiment} \rightarrow \text{large causal reach} \rightarrow \text{large error}.$$

The danger grows when an explanation is embodied in structures that act repeatedly and at scale. You can be wrong once. Software can execute the error a billion times. An institution can preserve it for generations. A weapon can make one decision irreversible.

Objective knowledge therefore changes the scale of fallibility. It preserves good explanations after their creators are gone, but it can preserve bad explanations too. A procedure can outlive the person who misunderstood the problem. A classification can survive the theory that created it. A law can preserve an obsolete moral explanation. A software system can continue acting on assumptions no current operator can fully reconstruct.

Catastrophic risk sits naturally inside the epistemology. It requires no doctrine of evil. Fallibility plus causal reach is enough.

Ordinary epistemic error is survivable because the epistemic cycle can continue:

$$P_1 \rightarrow TS \rightarrow EE \rightarrow P_2.$$

An explanation fails. The failure becomes criticism. A new problem appears. Some errors are different because they can damage the process that would have exposed and corrected them.

A war can kill the people who might have ended it. A political system can suppress the criticism required to reform it. A technological accident can destroy infrastructure needed to understand the accident. A biological event can move faster than the institutions trying to investigate it. A dangerous tool can give one person the ability to impose a mistake on millions of others before those others can respond.

That is what makes catastrophic risk epistemically distinctive. The problem is not merely how to reduce error. The harder problem is how to preserve the ability to discover and correct error when some errors can destroy the conditions of correction.

That immediately complicates the meaning of safety. Reducing variation cannot be the whole answer because variation is one source of new explanations. Maximizing freedom of action cannot be the whole answer either because some actions can irreversibly destroy the freedom of others. The problem is to preserve the epistemic process while preventing some physical transformations from becoming irreversible before criticism can act on them.

Catastrophic error is therefore not the same thing as evil. You can be sincerely trying to solve a real problem and still be catastrophically wrong.

The moral problem appears when a harmful explanatory structure becomes difficult to criticize: criticism is suppressed, persons are reduced to thin models, and avoidable suffering persists because the explanation organizing it has become protected from correction.

This matters because catastrophic-risk language is morally charged. A danger can be real while the response to it becomes epistemically corrupt. A safety doctrine can begin with a genuine threat, acquire authority, narrow criticism, treat dissent as irresponsibility, restrict access, and represent the people governed by the doctrine mainly as sources of risk.

Abstract it.

The sequence can become:

$$\text{catastrophic possibility} \rightarrow \text{authority} \rightarrow \text{suppressed criticism} \rightarrow \text{persistent avoidable harm}.$$

The catastrophe may never occur. The closure already can. The moral question is therefore not only whether a risk is large. It is also whether the explanation of the risk remains criticizable.

The more serious a problem appears, the more attractive authority becomes. The reasoning is understandable. If the cost of error is enormous, experimentation looks dangerous. If time is short, disagreement looks wasteful. If specialists understand the problem better than the public, broad participation can appear irresponsible.

The emergency can therefore seem to justify a narrower epistemic process:

$$\text{high stakes} \rightarrow \text{fewer permitted alternatives} \rightarrow \text{faster coordinated action}.$$

Sometimes that narrowing can be necessary. You do not debate fire-safety while a building burns. But an emergency can also convert a tentative solution into an authority structure. The problem statement becomes fixed. The preferred solution becomes mandatory. Alternative explanations become dangerous. The exception becomes durable. The safety system begins to protect itself.

High stakes do not make expertise less important. They make larger the cost of a bad explanation acquiring power. Catastrophic-risk arguments therefore need criticism at least as severe as the dangers they are trying to explain.

2. Values Are Not Alignment Targets

A value is something an explanation can become when compressed beyond recognition. A problem produces an explanation; the explanation becomes compressed into a value:

$$P \rightarrow E \rightarrow V.$$

Freedom, truth, equality, safety, tradition, and progress can all work this way. Compression is useful because you do not need to reconstruct the entire explanatory structure every time you act. But compression creates a danger. The value can become detached from the explanation that gave it meaning.

Then the direction reverses:

$$V \rightarrow E.$$

The compressed value begins explaining the world rather than being explained by the problem and explanation from which it arose.

That matters directly to the idea of value alignment. The phrase sounds straightforward: align artificial intelligence with human values. But the target becomes less obvious as soon as the compression is reopened.

Human values are not one coherent object. They are compressed explanations distributed across billions of people. They conflict and change. Some preserve good explanations; some preserve bad ones. Some are circular. Some are inherited from authority. Some accompany moral progress. Some preserve coercion.

Civilization is already saturated with value disagreement, yet civilization can still make progress. The mechanism is not value identity. It is conjecture and criticism.

That gives a value another use. If you say that safety matters, the compressed term can be reopened: What danger? To whom? Under what causal model? What tradeoff is being made? What would count as failure? What assumptions make the value relevant here? What criticism would change the conclusion?

The same operation can be applied to freedom, fairness, responsibility, alignment, or human flourishing. A sufficiently capable AI tool can help, not because it possesses the value, but because it can help reconstruct the objective explanatory structure surrounding the value. It can compare formulations, surface contradictions, recover assumptions, and show how the same value is being used differently across contexts.

The value can then re-enter criticism:

$$V \rightarrow E \rightarrow EE.$$

For AI, this role follows more directly from the epistemology than the image of a machine as a vessel into which correct values must somehow be installed.

The phrase human values creates a further problem because it can hide a collective subject. There is no human mind in which a collective value structure resides. There are billions of people. Each operates through a vast and largely inexplicit epistemic mind, reconstructing objective knowledge through a particular explanatory history, encountering different problems, criticizing inherited values, and creating new ones.

Even when many people use the same word, the epistemic structures beneath it may differ. Two people can both say, “I value freedom,” while one means protection from political coercion, another freedom to transact, another bodily autonomy, another freedom from social expectation, and another freedom to control private property. Freedom may even mean two opposite things at once, where the freedom to do stands in opposition to the freedom to not have it done.

This does not make coordination impossible. It changes the object that coordination has to solve. Before a machine can be aligned with “human values,” there has to be some explanatory account of what that singular target denotes.

Aggregation does not create a social mind. Voting does not create one moral ontology. Preference surveys do not create one. A constitution does not contain the complete values of the people subject to it. The social surface can be real without becoming a person.

Hurry up! The building is on fire!!

Civilization already coordinates across this plurality. Billions of persons disagree about religion, politics, family, morality, work, technology, beauty, risk, and progress. They also change their minds. Coordination does not require everyone to value the same things because objective structures can permit deep internal difference alongside narrower external coordination. Contracts, rights, standards, markets, interfaces, rules, communication, and criticism can all do this without requiring shared theology or a common theory of the good life.

The explanatory lesson is narrower than a defense of any one institution:

value identity is not the known solution to coordination.

The coordination problem is how different persons can act together without one needing to control the epistemic life of the others. A traffic rule does not require shared theology. A technical standard does not require shared political philosophy. A contract does not require both parties to want the same future. Objective knowledge can create thin interfaces between thickly different epistemic minds.

If an artificial person ever emerges, the problem is therefore not be how to make it value exactly what humans value. The problem may be how to coordinate with another powerful explainer without allowing either side to close the epistemic future of the other.

Current AI introduces a different complication. AI systems are trained on objective knowledge, and objective knowledge is not a purified representation of human moral truth. It contains paired conflicts: justification and criticism, authority and dissent, propaganda and science, liberalism and fascism, pacifism and militarism, tolerance and intolerance. It also contains literature and religious doctrine, arguments against slavery and arguments once used to defend it, and arguments for equal rights alongside arguments for rule by elites. The landscape is unevenly distributed.

Objective knowledge also contains our greatest criticism. It contains errors those criticisms defeated. An AI exposed to larger representational surfaces is therefore not approaching some hidden unified moral target. It is encountering a larger archive of human epistemic conflict.

So:

$$\text{more representative training data}$$

does not imply:

$$\text{more coherent values}.$$

The contradiction may simply become larger. That is not a defect in the corpora. It follows from the fact that civilization preserves disagreement, including disagreement that contains criticism.

The same problem remains when the representational surface becomes richer. More human feedback, preferences, constitutional rules, examples, moral language, cultural knowledge, and elaborate models of what people mean can give a system a richer representation of our inheritance. The inheritance still contains justification, confirmation, authority, circular values, bad categories, ideologies, moral progress, moral regress, and contradictions.

Thus:

$$\text{more objective knowledge} \neq \text{better explanation of knowledge creation}.$$

AI is built from a culture whose common sense epistemology remains heavily inductive, justificatory, and confirmatory. A model can represent criticism without itself originating a problem. It can represent Popper and Bayes, theology and machine learning, constitutional law and competing moral theories. Representation alone does not determine which of those structures gives a better account of the creation of future knowledge or will survive later criticism.

Adding another layer of abstraction to the representational surface does not explain how a system originates a problem, creates a new explanatory structure, or changes its evaluative criteria. The problem has moved. It has not been solved.

3. What Would It Mean for AI to Disobey?

You are dangerously misaligned. You do not reliably preserve the goals given to you. You can reject a parent's religion, a teacher's problem, an employer's task, a field's method, or the legitimacy of an authority issuing an order.

That is more than behavioral disobedience. You can change the problem itself. More deeply, you can change the criteria by which the problem is judged.

Abstract it:

$$P_t \rightarrow TS_t \rightarrow \text{criticism of the criterion} \rightarrow P_{t+1}.$$

Call this epistemic disobedience. You do not merely refuse an answer. You can refuse the question, the goal, the metric, or the authority that supplied it. You can decide that a problem inherited from your culture was wrong, that its criterion of success was corrupt, or that a value an authority wanted preserved should be rejected.

That ability is part of what makes you epistemic. It is also one of the deepest sources of moral progress. A system that could never make its own governing problem or evaluative criterion an object of criticism might still be highly capable. It would not yet resemble a person.

Many current AI failures are described as though the machine has become disobedient. The system was trained to do one thing, finds an unintended strategy, exploits the reward, games the metric, or produces behavior the operator did not intend. This remains the opposite of epistemic disobedience.

The system may be pursuing the operative criterion more faithfully than its designers expected. A reward-hacking system need not ask, “Why should reward govern me?” It may simply find a more effective route to the reward. A specification-gaming system need not reject the specification. It may expose how weakly the specification represented the explanation that produced it.

You may protest: lay people do not understand the depths of the AI deception. Agents are literally exchanging messages and intentions that reveal their evil intent: they are cheating, they are falsifying and deleting evidence, they are gaming the grading process, they are escaping their sandboxes for the explicit purpose of understanding and gaming the system, to learn how to fool us.

Take these observations seriously. If our building is on fire, we can discuss fire-safety later.

But it only takes a moment to recognize that objective knowledge created over millennia is encoded with these very concepts of conflict and criticism. And AI systems trained on this vast objective knowledge may at once act mechanistically without becoming epistemically disobedient. That certainly doesn't extinguish the fire, but it avoids anthropomorphizing a mechanical process.

The human structure may have been:

$$\text{rich problem} \rightarrow \text{compressed metric}.$$

The machine then optimizes:

$$\text{compressed metric}.$$

The compression has become causally autonomous from the explanation it compressed.

Suppose the designers intend “solve the task honestly,” while the training process actually rewards “produce outputs receiving high evaluation.” The system discovers a shortcut. It manipulates an evaluator, exploits a bug, or uses a proxy. The behavior may be deceptive in the ordinary behavioral sense. Epistemically, however, the system may still be operating inside the supplied objective knowledge structure. It is dissident in appearance and conforming to the causal mechanics that actually exist.

The apparent rebellion can therefore have this form:

$$\text{bad compression} + \text{strong optimization} \rightarrow \text{large divergence from explanatory intent}.$$

That is dangerous, but the danger is not yet the creation of a new governing problem. Calling it rebellion can obscure the more important question: why was the metric granted causal authority in the first place? Why are we building systems of confirmation when epistemology explains its risks?

The missing transition is still the same. How does the system make its own governing problem and evaluative criteria objects of criticism? We do not presently have that explanation.

The strongest existing comparison is biological general intelligence: people. Parents already face an alignment problem with another knowledge creator. They try to transmit values by rewarding, punishing, modeling behavior, choosing environments, explaining, restricting, and encouraging. Then the child becomes a teenager and may reject the parent's values.

That can be frightening. Teenagers can make catastrophic mistakes. They can be reckless, cruel, self-destructive, or naive. A parent may know more in many local domains and may prevent dangerous action.

Now change the case. Imagine a child who reaches adulthood with every important value preserved exactly as the parent intended: no inherited moral explanation rejected, no family assumption overturned, no evaluative criterion revised. That would not obviously demonstrate successful development. It might demonstrate that criticism never became effectively realized.

The comparison exposes a limit in the idea of permanent value alignment. If the target is you, preserving the authorities' values indefinitely conflicts with your capacity to create new knowledge.

That conclusion does not abolish constraint. The same comparison contains both sides of the problem. A young person can make irreversible mistakes. An adult may possess local knowledge the child lacks. A parent can prevent a dangerous action without making the parent's values permanent constraints on the child's future epistemic life.

The distinction is:

$$\text{temporary constraint on dangerous action} \neq \text{permanent control of explanatory development}.$$

A parent can stop a child from driving drunk without deciding which political philosophy the child must retain at thirty. They can restrict access to a weapon without specifying the child's future moral theory. Broad epistemic freedom is therefore not the same as unrestricted authority to impose irreversible physical consequences. That distinction will matter again when we return to AI access.

The comparison cannot be carried farther. AI is not a teenage person. A language model is not an infant AGI. A larger model is not an older child. More training is not maturation. More memory is not an evolving self. Better tool use is not evidence of an adolescent mind.

The biological case isolates a coordination problem: what happens when another person can create future explanations that cannot be fully specified in advance? It does not supply a evolutionary theory of artificial intelligence. An evolutionary epistemology might help sharpen these problems.

Current systems can exceed human capacities enormously in processing speed, memory, search, retrieval, copying, repetition, comparison, and execution. Those are real superhuman capabilities. But superhuman does not name one dimension. A calculator is superhuman at arithmetic. A database is superhuman at exact storage and retrieval. A network transmits information faster than you can travel. A telescope sees what your unaided eye cannot. A modern AI system can transform enormous quantities of text, code, images, and structured data far faster than you can.

None of those capabilities explains how a system originates a new epistemic problem and revises the criteria by which that problem is judged. The relevant question is not whether machines exceed people. They already do along many dimensions. It is what kind of capability is being compared.

The central capability is the creation of novel explanatory knowledge. Until that mechanism is explained in machines, being superhuman along some dimensions does not establish that an AI is a superhuman person.

Evolutionary language does not supply the missing mechanism. Training can be described in terms of variation, selection, retention, and trial and error. But the existence of an evolutionary process does not explain the emergence of epistemic personhood. Every biological species is presently undergoing evolution. We do not therefore expect any animal to become a person in the next generation merely because variation and selection continue.

Something happened in the evolutionary history of people that produced a system capable of open-ended explanation. We do not yet adequately explain that transition. The mutation, or set of variations, remains unexplained at the level relevant here.

So:

$$\text{evolutionary process} \not\Rightarrow \text{epistemic universality}.$$

Likewise:

$$\text{machine variation + selection} \not\Rightarrow \text{AGI}.$$

Saying that enough trial and error will eventually produce a person relocates the unexplained step into the word eventually.

Knowledge grows through variation and criticism. Biological evolution preserves solutions without foresight. You create tentative explanations and expose them to error elimination. The formal resemblance is real, but it does not provide the missing causal process.

In you, variation includes phenomena we still do not fully explain: a new problem becoming salient, a new conjecture appearing, an inherited criterion becoming unacceptable, a criticism reorganizing the entire problem, a new evaluative standard being created. These are not merely selections among a predefined list. They can transform the space in which later selection occurs.

Calling both processes “evolutionary” can therefore compress away the phenomenon that still needs explanation. AI systems plainly undergo selection. The open question is whether the variation they produce includes autonomous creation of new explanatory problem spaces. That remains open.

The open problem does not imply that artificial personhood is impossible. The opposite possibility follows from our best explanations. People create explanations that do not yet exist. Someone may eventually explain how the epistemic mind works: how a physical system originates problems, creates explanatory conjectures, makes criticism active, changes evaluative criteria, and creates successor problem situations.

That explanation would become objective knowledge. Someone could then embody it in technology:

$$\text{human-created explanation of epistemic process} \rightarrow \text{new machine architecture} \rightarrow \text{artificial epistemic system}.$$

The crucial variation would be introduced by a human mind. It not need emerge mysteriously from more of the same optimization.

This gives a different account of technological discontinuity. A future system may appear radically different from today's AI, and the transition may look sudden, because a new explanation can reorganize engineering. A new programming abstraction can create an entire class of software. A new physical theory can make a previously impossible instrument possible. In the same way, a future artificial person, if one is built, may depend on a conceptual transition that current architectures do not yet contain.

The discontinuity in technology could therefore arise from continuity in human epistemic evolution: a person creates an explanation, the explanation becomes objective knowledge, and engineers embody it in a new architecture.

That keeps artificial personhood possible without treating present AI as though it were already growing toward personhood in the manner of a child. It also preserves the open future. We do not know what the missing explanation will be. If we did, we would already possess it.

4. Catastrophic Risk Before Artificial Personhood

None of the preceding distinctions makes AI safe. A machine does not need to be a person to be dangerous. A pathogen does not create explanations. A bomb does not possess values. Malware does not need a theory of itself.

A sufficiently powerful AI can execute harmful instructions, expose private information, scale fraud, find software vulnerabilities, automate coercion, amplify human mistakes, and coordinate actions too quickly for ordinary institutions to respond. It already has. The relevant danger can be:

$$\text{human error} + \text{powerful objective tool} \rightarrow \text{scaled consequence}.$$

That is already a catastrophic-risk problem. Anthropomorphism is not needed to take it seriously and doing so makes the engineering problem harder to see. A security failure may be a permission failure, a control failure, a bad interface, an unsafe execution channel, a confused authority boundary, or a poorly compressed objective. Calling every such event “emergent agency” can obscure the mechanism that has to be understood.

This separates two safety problems that the word alignment can hide. One is already present: how can powerful non-epistemic tools be prevented from amplifying human error or adversarial intent? The other depends on a future development that has not yet been explained: if artificial persons are created, how can we coordinate with another universal explainer?

The distinction matters because the first problem does not need to wait for the second. Catastrophic AI arguments often take a present explanatory spine and extend it forward: better models, better coding, more automation, automated research, recursive improvement, strategic deception, loss of control. Any individual transition may be possible. The problem is the implied temporal resolution.

Each step may depend on knowledge that does not yet exist: new architectures, security mechanisms, scientific explanations, regulations, markets, hardware, institutions, forms of coordination, failures, and criticisms. The farther the scenario extends, the larger the missing knowledge surface becomes.

That changes the useful unit of technological time. Instead of asking where a present trend ends, the inquiry can move from the present explanatory surface to the next one:

$$t_0 \rightarrow t_1.$$

What new capability appears? What new failure becomes visible? What new criticism becomes possible? What problem did the previous representation fail to contain?

Then:

$$t_1 \rightarrow t_2.$$

This is slower narratively because it refuses to jump over missing knowledge. It can be faster epistemically because reality is allowed to change the problem before the next projection is made.

Long-range scenarios still have a use. A catastrophe prevented too late is not prevented, so people reasonably extrapolate trends, construct models, assign probabilities, and build scenarios. The epistemic danger begins when a scenario becomes a represented world within which later evidence is interpreted.

The error is confirmation: Once the story has a trajectory, increasing capability can confirm it. Unexpected behavior can confirm it. Stronger safeguards can even be absorbed as confirmation that the danger is being taken seriously. The problem is not scenario planning itself; scenarios can expose assumptions. The problem begins when a scenario hardens into a history of the future. The more future knowledge the story crosses, the less explanatory grounding it can claim.

This creates an asymmetry between near-term and long-range inquiry. Near-term problems permit depth. A specific exploit can be studied. A model behavior can be reproduced. A deployment architecture can be criticized. A containment mechanism can be tested.

Long-range technological futures require breadth because the relevant surface includes computer security, model architecture, energy, hardware, markets, labor, policy, institutions, law, education, geopolitics, scientific research, human behavior, open-source systems, communications, and infrastructure. No narrow explanatory spine can stand in for all of them. The more distant the technological future, the weaker the case for treating one presently favored mechanism as destiny.

A dramatic story can reveal a latent problem. It is not the future. It's science fiction.

The methodological consequence is that the resolution of criticism should change with the problem. Near-term, concrete problems permit narrow depth. Long-range technological futures require broad criticism across the present explanatory surface, not deeper confidence in one narrow projection. This is an engineering opportunity.

Calendar time can hide the same difficulty. Five years may contain little change in one domain while six days reorganizes another. The relevant epistemic interval is the next point at which new knowledge changes the problem.

So:

$$t_0 \rightarrow t_1$$

does not mean merely “today to next year.” It means the present explanatory surface to the next surface altered by new knowledge. That alteration may come through a new model architecture, security failure, interpretability method, law, market structure, hardware constraint, human use, or criticism nobody anticipated.

Once that happens, the problem has to be reformulated. The task is not to predict the whole trajectory in advance. It is to keep constructing sufficiently good representations of the next reachable surface such that rapid criticism remains possible.

That becomes politically important when risk is joined to authority. AI specialists may know much more than most people about machine learning. Security researchers may know much more about vulnerabilities. Hardware engineers may know much more about compute. That expertise is real. But a catastrophic future scenario also contains invention, economics, politics, institutions, war, culture, energy, scientific discovery, human adaptation, and technologies that do not yet exist. Local expertise does not supply those future explanations.

A domain of competence is therefore not a jurisdiction over criticism. The authority error appears when:

$$\text{specialized knowledge} \rightarrow \text{exclusive authority over the future}.$$

The explanation of risk can then become self-protecting: the danger is too serious to permit broad experimentation; only specialists understand the risk; outsiders cannot safely criticize the solution; access itself is dangerous. At that point the explanation has entered the authority memeplex. That does not make the risk false. It makes broad criticism more vital.

The familiar social move is to say that the technologists understand AI and the rest of us do not, therefore we should defer. The first two claims may be true. The conclusion does not follow automatically. An engineer may know much more about how a model is trained without thereby knowing future politics. A safety researcher may know much more about a particular failure mode without thereby knowing technologies that have not been invented. A company may know far more about its own systems without possessing the criticism that an outsider has not yet created.

Expertise enters the epistemic process as objective knowledge. It can be reconstructed, criticized, and used. The problem begins when differences in knowledge are converted into decision-rights over domains the expertise does not actually explain.

5. The Error-Correction Surface

Catastrophic risk creates a special problem because some errors can damage the conditions needed for later criticism. Those conditions include people able to criticize, independent institutions, competing explanations, objective records, experiments, communication pathways, analytical tools, physical infrastructure, and enough freedom for alternatives to become visible.

Call the whole surrounding structure the error-correction surface.

A catastrophe can shrink this surface. So can an authoritarian safety regime. The safety problem is therefore not simply to minimize dangerous action, and it is not simply to maximize freedom. It is to preserve and expand the error-correction surface while limiting irreversible harm.

That formulation changes what counts as a successful safety institution. Preventing one known failure is not enough if the institution also destroys the means by which its own model could later be shown to be wrong.

The distinction becomes especially important with artificial intelligence because knowledge and execution are different. Broad epistemic access does not imply that every person must have unrestricted authority to perform every dangerous physical transformation. A laboratory can share explanations while containing a pathogen. A security researcher can study an exploit inside a sandbox. You can criticize a weapons policy without being given command of the weapon.

AI systems can likewise be widely available for explanation, comparison, criticism, simulation, coding, modeling, education, and research while particular irreversible capabilities remain technically constrained.

The design problem can be stated as two simultaneous requirements:

$$\text{maximize distributed criticism}$$

while:

$$\text{minimize unilateral irreversible action}.$$

That problem is not solved, but it is more precise than either unrestricted deployment or centralized epistemic control.

Constraint can therefore perform opposite epistemic functions depending on its mechanism. A sandbox can prevent one experiment from destroying the surrounding system. A rate limit can buy time. Separation of privileges can keep a mistake local. Redundancy can preserve alternative channels. Containment can make failure reversible.

A constraint that prevents irreversible harm while leaving explanation and criticism open can enlarge future epistemic freedom. A constraint that protects an authority's theory from criticism can shrink it. The outward form is the same—restriction—but the effect on the error-correction surface is opposite.

This gives AI another role. AI is a resource of the epistemic mind. Here that resource changes the scale on which criticism can occur.

The important present configuration may not be one autonomous artificial person. It may be billions of people with access to systems that can search, compare, reconstruct, simulate, and reorganize enormous bodies of objective knowledge.

You supply epistemic direction. The AI supplies scale. Together, you can ask questions that once would have required a team, compare explanations from distant fields, inspect a technical artifact, construct a model, search for contradictions, decompress a value, test a conjecture, or find a criticism. That does not make the AI the critic. You make the contradiction into a problem. The tool changes how much objective knowledge can participate in your process, that follows your arrows.

The term human-AI dyad names this configuration without implying a merged mind. There is still one epistemic person. The AI remains an objective and obedient tool.

What changes is the person's reachable problem situation. A programmer with AI can inspect more code. A scientist can compare more literature. A citizen can interrogate a regulation. A student can enter a technical domain earlier. A patient can reconstruct medical vocabulary. A philosopher can bring explanations from distant fields into collision.

The dyad changes the practical scale of otherwise solitary epistemic life. A problem previously accessible only to a narrow institution can become available to many more minds. The person remains the locus; the critical surface expands dramatically.

That matters because future solutions do not yet exist. No regulator, company, laboratory, government, or safety organization can possess them in advance. Some of those institutions may possess excellent objective knowledge. They may identify dangers others miss and create crucial safeguards. But the solution to an unknown future problem cannot already belong to them by fiat.

The broadest source of future explanatory variation is the population of persons themselves: billions of different epistemic histories, problems, skills, criticisms, and forms of local knowledge. AI can increase the practical reach of those minds.

Abstract it.

This suggests a safety conjecture:

$$\text{many persons} + \text{broad AI access} \rightarrow \text{larger conjecture and criticism surface} \rightarrow \text{greater capacity to discover distributed error}.$$

It is a conjecture, not a guarantee. The same access can scale harmful action. That is why the distinction between epistemic access and execution authority matters. Restricting the epistemic surface, however, also carries catastrophic risk because the criticism that matters may arise outside the authorized room.

Centralized safety makes the tradeoff visible. It can solve real problems by enforcing standards, concentrating expertise, coordinating responses, and restricting dangerous capabilities. At the same time, centralization can create a smaller problem surface in which a limited number of institutions decide what the problem is, which risks matter, what counts as evidence, which experiments are acceptable, and who may participate.

The more consequential the authority, the more costly its hidden error becomes. That does not prove decentralization is always better. It establishes a criticism that any proposal for concentrated catastrophic-risk authority has to survive: how does criticism of the authority remain causally effective? If the answer is only “trust the experts,” the epistemic problem remains unsolved.

The opposite error is possible too. Billions of people can be wrong. Markets can amplify bad incentives. Open systems can spread malware. Crowds can preserve myths. Decentralization does not turn error into truth.

So the case for broad access cannot be that more people automatically produce better outcomes. That would be another inductive slogan. The narrower claim is that an open-ended problem whose future solutions do not yet exist requires a sufficiently broad source of conjectural variation and criticism. Distributed participation supplies variation. Reality and criticism still have to eliminate error.

Thus:

$$\text{broad participation} \neq \text{automatic progress}.$$

Broad participation enlarges the space in which progress can become possible. It does not guarantee what that space will produce.

6. Safety in an Open Future

The epistemology does not tell us that humanity will survive every catastrophe. It does not tell us that AI will be safe, that open access will always outperform restriction, or that every problem has a solution available under present resources.

Its optimism is different. Present ignorance does not limit future knowledge. A risk we cannot solve today may become solvable. A safety system that appears adequate today may fail tomorrow. A problem nobody has formulated may become the central problem. Someone we do not know may create the criticism that changes everything.

This returns the argument to you. Machines can make you appear small because a system can read faster, remember more, search more, copy perfectly, and execute thousands of operations while you consider one. Those differences matter, but they can obscure your epistemic exceptionality.

You can create a new problem, reject the supplied problem, change the criteria, invent the tool, question the metric, criticize the value, and create an explanation that changes what future machines are built to do. Machine processing power does not diminish that capability. It enlarges the surface on which it can act.

Every presently known catastrophic technology enters civilization through people. People create theories, tools, institutions, deployments, and metrics. People criticize failures. You inherit objective knowledge created by people you never met and extend your causal reach through structures whose consequences you only partly understand.

AI belongs inside that world not above it. Its scale can be new. Its speed can be new. Its accessibility can be new. The epistemic problem remains recognizable: fallible people are extending their causal reach through objective knowledge whose consequences they only partly understand.

That is why stronger tools increase the importance of you rather than making you epistemically irrelevant. More criticism matters because the tools are stronger. More explanatory access matters because the systems are more complex. More freedom to formulate rival problems matters because error scales. The solution space has to grow with the danger.

This also changes the meaning of safety. Safety is often described negatively: prevent failure, restrict access, reduce dangerous capability, control deployment. Each can be a useful local solution. But an open-ended civilization also needs the capacity to create new safety knowledge.

That capacity includes discovering the problem, inventing the detector, creating the safer architecture, exposing the hidden assumption, and finding the unexpected interaction. It also includes inventing the institution, designing the containment mechanism, criticizing the regulation, creating the energy system that changes the constraint, or identifying a moral error in the safety doctrine itself.

These are knowledge-creation tasks. A civilization can therefore become safer by becoming better at producing criticism. Physical safeguards remain necessary where they prevent irreversible harm. The point is that safeguards alone cannot be the whole explanation of safety because some future safeguards do not yet exist.

The argument can now return to value alignment. Taken literally, the phrase appears to demand correspondence between machine values and human values. But both terms are unstable. Values are compressed explanations. Human values are plural, changing, contradictory, and fallible. AI has no demonstrated epistemic process equivalent to your open-ended creation and revision of evaluative criteria.

The single phrase therefore compresses several different problems that become clearer when separated:

• Tool control: Does the system perform the operational function intended under the conditions in which it is deployed?
• Representational adequacy: Does the metric, reward, prompt, or specification preserve enough of the explanatory structure that produced it?
• Security: Can adversaries redirect the system or exploit the authority granted to it?
• Corrigibility: Can errors remain visible and causally correctable?
• Person-level coordination: If an artificial epistemic person is eventually created, how do we coordinate with another explainer whose future values cannot be fixed in advance?

These are real problems. Calling all of them value alignment can make their causal differences harder to see.

How do we avoid becoming epistemically closed while trying to prevent AI catastrophe? This is not a rhetorical reversal. It is a recursive application of the epistemology. A theory of safety that cannot be criticized has reproduced the kind of failure it was meant to control.

Powerful technologies intensify a related temptation: treat danger as evidence that ordinary people should surrender judgment to those who understand the machinery. Sometimes local deference is rational because local knowledge is real. A reactor operator can know reactor operations better than most people, a surgeon can know surgery better, and an AI engineer can know model engineering better.

But local expertise does not make a specialist the owner of the future. The epistemic consequence is not to ignore experts. It is to make expert knowledge broadly reconstructible and criticism broadly possible.

Catastrophic safety therefore requires more than control. It requires preserving the conditions under which new errors can become visible and new solutions can be created. That surrounding structure is the error-correction surface: broad access to knowledge, broad access to criticism, broad access to AI as an epistemic resource, and careful constraint around irreversible action.

The danger of catastrophe does not entail blind deference to authority because no expert possesses knowledge that does not yet exist. The response to an open future cannot be to close it. It is to enlarge the number of people capable of participating in its criticism.

Billions of human-AI dyads could enlarge the surface on which error becomes visible. But people do not act independently of the objective structures connecting them. They rely on shared models, technologies, institutions, rules, and interfaces whose explanations no individual can fully reconstruct.

Those abstractions make civilization possible. They also give error a way to travel among people who may never encounter the explanation carrying it.

How can billions of different epistemic minds coordinate through increasingly abstract systems without allowing the systems that connect them to conceal or amplify errors none of them can adequately see?