Microsoft CEO: The superintelligence era cannot rely on AI "black boxes"; external control and audit systems are needed

PANews reported on October 11 that Microsoft CEO Satya Nadella published a post stating that as the capabilities of superintelligence systems rapidly improve, the industry needs to rebuild its AI trust architecture and cannot shift responsibility for AI execution behavior onto model providers.

Over the past few decades, when traditional software systems were widely deployed, people could trace the specific code paths corresponding to program behavior. But today's frontier AI models, although their capabilities have surpassed traditional software systems, cannot clearly attribute model outputs to specific training data or model weight configurations. Enterprises are currently exposing complex AI agents to sensitive data and granting them the ability to execute critical tasks. Therefore, superintelligence systems cannot be treated as incomprehensible "nested black boxes" whose recommendations are simply accepted or rejected; instead, we need to build controllable systems that can observe behavior, test boundaries, and restrict actions at any time.

In terms of governance principles, Nadella proposed that superintelligence systems need to have:

- Model diversity: avoid making a single model the sole dependency for critical decisions;

- Comprehensive observability: all important AI behavior should leave tamper-proof, human-readable records;

- Continuous verification capability: continuously test the system's performance in the face of failures, attacks, and anomalies;

- Independent control and audit: a model should not simultaneously control system behavior and verify the evidence of its own behavior;

- Risk isolation mechanisms: assume by default that the model may fail, and provide the ability to pause or shut it down at any time;

- Incident disclosure mechanisms: promptly disclose causes and improvement measures when failures or security incidents occur.

Nadella also emphasized that chain-of-thought (CoT) transparency is an important foundation for AI governance, but CoT transparency alone is not enough, because it is still impossible to guarantee that model outputs always have reliable interpretability. In the future, the most trustworthy superintelligence systems will not be "the most trustworthy models," but rather systems that can still operate safely even when the model cannot be fully trusted.