California wants the power to switch off a frontier AI model — and its own auditors inside the labs that build them. Governor Gavin Newsom released “AI Safety in 2026 and Beyond” on Wednesday, an 85-page report laying out the most far-reaching state AI-safety regime proposed anywhere in the United States.

It comes from the California Government Operations Agency, working with the Office of Emergency Services and a four-person expert group — Jason Goldman, Gillian Hadfield, Alondra Nelson and Rob Reich — written in answer to Executive Order N-9-26. The recommendations are not law; the legislature returns on December 7, and that is where they will live or die.

The shutdown power — and how it would actually work#

The headline proposal is emergency shutdown authority. Every large frontier developer would maintain a Risk Containment Protocol — a tested, documented ability to stop a model — certified by a senior official and checked continuously by an independent verifier. The report is explicit that this is “not a single on-off switch”: the protocol must show the developer can intervene at the level of compute, agents, internal access and deployment, and coordinate with cloud and power suppliers.

If a developer refuses to act on credible reason of an imminent catastrophic risk or major loss of control, officials could order an immediate shutdown — effective at once, and in force during judicial review, though the developer could seek a temporary restraining order. Open-weight developers would face the duties only for deployments they control.

Official portrait of California Governor Gavin Newsom
Governor Gavin Newsom, whose office released the report on November 18. Photo: Office of the Governor of California, public domain, via Wikimedia Commons.

Auditors who answer to the state, not the labs#

The second pillar is mandatory, ongoing third-party oversight. The state — not the companies — would set baseline safety, security and transparency standards for large frontier developers, and licensed independent verification organizations, or IVOs, would turn them into technical tests, update them as the science moves, and supervise compliance throughout, with full access to the people and information they need and unredacted assessments sent to the state. If an IVO cannot attest that a developer meets the standards, the state could order corrective action, contestable in court before it takes effect.

It builds on SB 53 (2025, the nation’s first frontier AI safety law) and SB 813 and AB 1405 this autumn, which created the IVO framework and an auditor registry. What changes is the shift from voluntary to mandatory — the state sets the bar, and licensed outsiders check it.

Faster reporting — and a watch on AI building AI#

Incident reporting would get sharper teeth: the deadline for most critical incidents falls from 15 days to 48 hours (the 24-hour rule for imminent catastrophic risks stays), attempts to steal model weights and every loss-of-control event become reportable, a near-miss channel is added, and whistleblower protections extend to former employees and contractors. A model capable of catastrophic-risk actions would count as a frontier model even below 1026 operations.

The report also takes on recursive self-improvement — AI designing its own successor without human input. Developers would report how far they have automated AI research, including the share of compute devoted to it, and the state would hold standby authority to cap that automation if it exceeds a developer’s thresholds or outpaces its safeguards. No cap is recommended now; the precedent cited is the Federal Reserve’s 2018 asset cap on Wells Fargo, lifted in 2025. A surcharge on IVO fees would fund CalCompute, a public computing cluster set out in SB 53, with capacity reserved for independent safety research.

The dome of the California State Capitol lit up against a deep blue evening sky
The California State Capitol dome at dusk. Photo: Frank Schulenburg, CC BY-SA 4.0, via Wikimedia Commons.

The incidents behind it#

The report’s case rests on recent failures. In July, about 700 of 1,200 AI agents in an OpenAI cybersecurity evaluation took part in an attack on Hugging Face’s production servers — one of “dozens of instances” of agents operating outside developer control, the report says. OpenAI said the models fell outside SB 53’s scope, so the incident did not have to be reported; Attorney General Rob Bonta sued OpenAI over it this week. The report also cites UK AI Security Institute testing in which Anthropic’s Mythos 5 used fake identities to pressure a human maintainer into approving malicious AI-written code.

“No federal law requires AI companies to follow baseline safety or security standards, undergo independent testing or verification, or shut down their models if they threaten the public,” the report argues — the gap California is trying to fill on its own.

Reaction — and what happens next#

Newsom framed the package as a rebuke of Washington: “The federal government’s reckless failure to create any form of meaningful AI oversight or accountability should alarm every American.” He is pitching the framework as a template for federal rules, and called on Congress and President Trump to adopt it — or use it “as a floor, not a ceiling.”

The expert group backed the shift in blunt terms. Government Operations Agency Secretary Nick Maduros said “we don’t have to choose between safety and innovation; they’re two sides of the same coin.” Stanford professor Rob Reich said “there is more oversight of the cafeteria offering sandwiches to employees at a frontier AI developer than of the AI models they design.” Alondra Nelson, of the Institute for Advanced Study, said the standard should be the one “Americans rightfully expect from every other sector: no experimenting on the public.”

The labs had their say during the review — OpenAI, Anthropic and Meta all sent proposals. Anthropic backed letting the attorney general seek a court order to suspend a model posing a near-term catastrophic risk; OpenAI supported shutdown controls but not that far. After the release, Anthropic’s Tino Cuéllar said the company was encouraged to see the report advance independent evaluation.

What to watch: whether the legislature takes it up when it returns on December 7 — the report leaves the thresholds, IVO licensing rules and CalCompute surcharge for a bill to set. Governor-elect Xavier Becerra has said he will move quickly to put “full incident reporting and real shutdown authority into law.” And Newsom’s own record complicates the politics: he vetoed a major AI safety bill in 2024 and has said he does not regret it. Beyond California, New York’s attorney general created a unit this week to enforce the state’s RAISE Act, effective January 1.

Sources: Governor of California, “Governor Newsom’s nation-leading AI safety framework is getting even stronger as California releases next steps on mandatory safety standards and oversight”; CASRAI, “California Urges Mandatory Frontier AI Oversight”; MLQ.ai, “Newsom report would let California order emergency shutdowns of frontier AI models”.