California Blinks: Governor Newsom Vetoes AI Bill Aimed at Catastrophic Harms

In an 11th hour decision, Governor Gavin Newsom has vetoed a bill widely seen as one of the most ambitious efforts to regulate the generative AI industry. SB 1047 was one of hundreds of bills floated around the country in the absence of meaningful action from Congress to address potentially catastrophic impacts of the software.

SB 1047 would have required developers to submit their safety plans to the state attorney general, who could hold them liable if AI models they directly control were to cause harm or imminent threats to public safety.

“The harms at issue in SB 1047 are harms that people are trying to perpetrate today,” state Senator Scott Wiener (D-San Francisco) told KQED. “The question is, are we going to allow large AI models to make it even easier and smoother for people to do things like shut down the electric grid or build a chemical weapon? It’s not science fiction at all.”

In his veto message on the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, Newsom made it clear that he’s not averse to regulating AI companies, noting that he’s already approved more than a dozen bills around specific risks of harm to democracy, online privacy, and critical infrastructure. But he criticized SB 1047’s limited focus on the most expensive and large-scale models, saying it “establishes a regulatory framework that could give the public a false sense of security about controlling this fast-moving technology.”

Wiener spent the last year in a pitched battle with some of the most prominent figures in the AI industry, including many who warned the measure could stifle the growth of the technology in California. Wiener said he amended the measure to reflect counsel from leaders in the AI space, including safety groups, academics, startups and developers like Amazon-backed Anthropic.

Jack Clark, co-founder of Anthropic, tweeted Sunday that, “While the final version of SB 1047 was not perfect, it was a promising first step towards mitigating potentially severe and far reaching risks associated with AI development.”

Wiener eliminated text that would have allowed California’s attorney general to sue AI companies for negligent safety practices before a catastrophic event has occurred, along with plans to establish a division within the California Department of Technology that would have provided oversight and enforcement.

Compared to the original language of the bill, what landed on the governor’s desk was substantially weaker, according to Gary Marcus, a scientist, entrepreneur and author of Taming Silicon Valley, a book highly critical of generative AI. “The bill was watered down,” Marcus said, adding he feels its value was primarily symbolic, and that Newsom’s decision signaled to Silicon Valley that it can “cause enormous chaos, and probably nobody’s going to make them fix it.”

As with other measures before Newsom, the governor had a month to consider whether to sign SB 1047 or veto it, and during that time, his office was lobbied heavily by industry insiders on both sides, as well as local congressional representatives and Hollywood celebrities.

“I don’t have the technical capacity to perfectly predict how [SB 1047] would have affected the AI industry,” said Thad Kousser, a political science professor at UC San Diego. “Many people in Sacramento don’t have that ability. Maybe they just decided to err on the side of caution, thinking, ‘Wow, there’s so many industry voices saying this particular bill is dangerous and could have a chilling effect.’ Not really knowing 100%, maybe the safer step is just vetoing,” Kousser said.

Wiener said the governor’s office did not engage with his office as the bill made its way through the state Assembly and Senate. “I personally met with some of the most vocal opponents of the bill, with the Andreessen Horowitz firm, with several of the Stanford professors who were opposing the bill, with the large tech companies that were opposing the bill. I also met with individuals and businesses that had constructive criticism of the bill. And we made significant changes to the bill in response to those constructive critiques.”

Those changes were not enough for many critics, including Congresswoman Zoe Lofgren of San José, who wrote, “Any AI risk framework should be based on empirical data and fit for purpose. I also believe this is an issue that should be handled at the federal level. Congress and the Administration are both moving on AI governance. I look forward to working with the Governor as we move forward.”

Lofgren is the ranking member of the House Science, Space and Technology Committee, which has in recent weeks moved nine bills forward, but all face an uncertain future in the House of Representatives. Nothing addressing the scope and scale of SB 1047 has passed out of any committee. But Lina Khan of the Federal Trade Commission has said that federal regulators are keen to use existing laws (PDF) to go after bad behavior in Silicon Valley and elsewhere.

Unlike the European Union and Colorado, both of which passed comprehensive laws, California lawmakers have largely focused on discrete bills addressing specific issues with generative AI. Gov. Newsom signed 17 of these bills this year, as he noted in his veto message, and California is among a host of states taking steps to regulate generative AI.

Given the inertia in Washington, D.C., most political analysts see the state level as the only hope for aggressive regulation of technology.

In the race for the White House this year, both Kamala Harris and Donald Trump have sought the support of Silicon Valley’s most powerful players. Newsom’s presidential ambitions are presumably on hold for the foreseeable future, but given the national profile of SB 1047, some have wondered if he might be loath to make enemies among those profiting from the rise of generative artificial intelligence.

“I think, like any good politician who’s ascended to the heights Gavin Newsom has, you’ve got to be thinking about ‘How will I be judged today, tomorrow, in 5 or 10 years,” said Professor Kousser. “That forward thinking has guided his decisions on many bills throughout his governorship. He’s been on the right side of history in many of the strong policy stands he has taken, as mayor and as governor — and he’s hoping that he’ll be on the right side of history on this one.”

Industry critics like Marcus argue something much more cynical: that the biggest of the big Silicon Valley companies have become so wealthy and powerful that few U.S. politicians appear willing to go up against them. “Existing laws didn’t anticipate the AI era, and so there are lots of gaps in those laws. We don’t really have clarity about what to do about copyright. We don’t have clarity about what to do with machines when they go out of control in some way.”