A bill aimed at regulating powerful artificial intelligence models passed California's legislature on Wednesday, despite outcry that it could kill the technology it seeks to control.

"SB 1047 -- our AI safety bill -- just passed off the Assembly floor. I’m proud of the diverse coalition behind this bill -- a coalition that deeply believes in both innovation and safety," said Democratic state senator Scott Wiener of San Francisco, who sponsored the bill.

The bill had faced a barrage of critics, including Democratic members of US Congress, who argued that threats of punitive measures against developers in a nascent field would throttle innovation.

It did, however, win the reluctant support of Elon Musk, who argued that AI's risk to the public justifies regulation.

"This is a tough call and will make some people upset," he said on Monday as he stated his support on X.

The bill, called the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, now goes to the desk of California Gavin Newsom for signature, though his position on the proposal is not known.

Newsom has until September 30 to sign the bill or put his veto.

Dan Hendrycks, director of the Center for AI Safety, said the bill provides "a workable path forward" in enforcing "safeguards to mitigate against critical AI risk."

The bill requires developers of large "frontier" AI models to take precautions such as pre-deployment testing, simulating hacker attacks, installing cyber security safeguards, and providing protection for whistleblowers.

In order to secure the legislation's passage in the home state of Silicon Valley, lawmakers made several changes to the bill, including replacing criminal penalties for violations with civil penalties such as fines.

But opposition to the bill remained, including from some powerful national figures.

"The view of many of us in Congress is that SB 1047 is well-intentioned but ill-informed," influential Democratic congresswoman Nancy Pelosi of California said last week, noting that top party members have shared their concerns with Wiener.

But Wiener argues that AI safety and innovation are not mutually exclusive, and that tweaks to the bill have addressed some concerns of critics.

OpenAI, the creator of ChatGPT, also opposed the bill, saying it would prefer national rules instead of a chaotic patchwork of AI regulations across the 50 US states.

At least 40 states have introduced bills this year to regulate AI, and a half dozen have adopted resolutions or enacted legislation aimed at the technology, according to The National Conference of State Legislatures.