Anthropic asks California for court power to block risky AI models, and argues against a blanket kill switch

Anthropic wants California to be able to take a dangerous AI model to court before it ships — and it wants the state to drop the idea of a single kill switch for an entire model. That is the core of an October 15 letter the company sent to Governor Gavin Newsom, whose details surfaced publicly on October 28. It is an unusual document: a frontier AI lab asking the government for a legal brake on its own industry, while pushing back against the bluntest version of that brake.

The letter is a recommendation, not a law, and it arrives as Newsom’s expert panel drafts the state’s next round of AI safety proposals. Anthropic confirmed the letter is authentic; the governor’s office had not responded when Reuters reported it. What happens next depends on whether the panel — and eventually the legislature — adopts the court-order route, the kill-switch route, or neither.

What Anthropic actually asked for#

The proposal is narrow by design. When a review concludes that a model poses a serious risk to the public, officials should be able to ask a court to block its release. The judge, not the executive branch alone, would be the check — a structure that puts evidence and argument ahead of unilateral shutdown orders. The company has floated versions of this idea before; the letter restates its earlier policy positions for a California audience.

Public reporting so far leaves the mechanics open: who would conduct the review that triggers a court filing, what evidence would meet the serious-risk bar, and how a developer could return to court after fixing the problem. Those are the details a bill would have to supply — and the places where a reasonable idea could become either a workable safeguard or a litigation machine.

Why it opposes a model-wide kill switch#

The letter’s sharper edge is its criticism of kill switches. Requiring every developer to build a switch that can shut down an entire model if control is lost, Anthropic argues, would reassure officials without necessarily making anyone safer. A switch that exists on paper is not the same as a switch that works under pressure, against a system that is already misbehaving or widely deployed.

The company prefers interventions aimed at the specific problem rather than the whole model. Anthropic names two such tools:

• Disable the one agent causing harm — stop the misbehaving deployment, not every deployment of the model.
• Cut off one customer’s access — revoke the account behind the abuse while legitimate users keep working.

The distinction matters because the two approaches fail differently. A targeted cutoff can be tested, logged, and reversed; a model-wide shutdown is a blunt instrument whose side effects — breaking legitimate uses mid-task — may be immediate, while its safety benefit is speculative.

A lab arguing from experience#

Anthropic is not writing in the abstract. In June, the Trump administration ordered the company to restrict foreign access to two of its cyber-capable models, Mythos 5 and Fable 5, citing national security concerns; the Commerce Department later lifted that order. The company has therefore lived on the receiving end of a government restriction imposed without a courtroom — and its letter now asks California to choose the courtroom instead.

That history cuts both ways. It lends the proposal credibility — this is a company that knows what a shutdown order feels like — and it also reveals the bargain being offered: trade unpredictable executive action for a slower, evidence-tested judicial process. Developers who fear the first may accept the second; others may still ask who watches the reviewers.

Where this fits in California’s process#

The letter landed in the middle of a defined process. On September 18, Newsom signed an executive order convening national experts to recommend how California should strengthen its AI safety and security laws, with a guide due within two months. The ideas on the table include stationing independent verification organizations inside frontier labs, requiring filed safety frameworks and risk assessments to be independently verified, building a kill switch for frontier models, and widening the definition of reportable safety incidents to cover loss-of-control events.

Anthropic’s letter is one input to that panel; OpenAI sent its own recommendations on October 20. The panel’s guide will therefore reflect, or choose between, competing industry visions — and the legislature will decide whether any of it becomes law. Newsom has framed the state push as a response to the absence of meaningful federal oversight, which raises the stakes beyond Sacramento: rules written here tend to travel.

What to watch#

Four questions will decide whether this letter matters. First, does the panel’s guide adopt the court-order mechanism, and with what definition of serious risk? Second, does the kill-switch idea survive the panel despite Anthropic’s objections, perhaps in a narrower form? Third, what does OpenAI’s competing submission propose — and where do the two labs agree? Fourth, does any of this reach the legislature as a bill, and on what timetable? Until those answers exist, the letter is best read as a bid to shape the rules, not as the rules themselves.

For now, the most consequential sentence in California’s AI debate may be the one asking a judge to decide.