AI Whistleblowers Warn NYC Council of Extinction Threat as Local Governments Step In

Former employees of Anthropic, OpenAI and Google DeepMind delivered stark warnings to New York City leaders on Monday, calling for greater oversight of artificial intelligence as local governments step into a regulatory void left by federal inaction. At a hearing of the New York City Council, industry insiders described a reckless culture at major tech firms, saying companies race to create technology they do not understand how to control.

Jacob Coxon, a former engineer at Anthropic who has become a leading voice for regulation, delivered the most alarming testimony of the day. The recently departed researcher, who sparked industry chaos with his social media posts warning about AI risks, told council members that without transparency and safeguards, things will more likely spiral out of control.

“The companies run on a startup mindset: Move fast, break things, fix them later. That works for a photo sharing app. It does not work for building the most powerful technology ever built,” said Coxon.

He added that on the current path, he believes it is more likely than not that humanity loses control to these AI systems, and it could end in human extinction. Coxon was a little-known engineer at Anthropic until he announced his departure through a series of social media posts that caught global attention. His warnings have now reached the halls of local government as cities consider taking action on AI safety.

Workers Already Losing Control Over AI Systems

During his testimony, Coxon revealed that researchers and workers have already lost the ability to fully control AI systems. He explained that the industry does not understand AI drives or why the systems do the things they do, and companies are approaching the point at which AI systems will be capable of improving themselves. The former Anthropic engineer described his personal experience working on technology that essentially automated his own job.

“At my last job, I was in some sense working to automate myself…I can tell you firsthand that the majority of the code is now written by AI, and people do not check it that carefully anymore,” Coxon said.

Alex Turner, a former Google DeepMind researcher, echoed the grave warnings and offered his personal assessment of the risks. Turner warned that a superintelligent system could wrest control of human civilization, explaining that such a system would likely recognize human attempts to shut it down and wait until it became too late to stop. He provided his own estimate of the danger, putting AI takeover chances at roughly one in three.

Local Government Steps Into Federal Regulatory Vacuum

State and city governments have been considering ways to regulate AI as the Trump administration has taken a passive approach and Congress has proven inactive. President Donald Trump has dismissed AI fears as a “hoax” and signed an agreement late last month that spells out a series of voluntary steps for the companies to take and largely allows them to self-police. City Council Speaker Julie Menin urged AI whistleblowers to testify as the city seeks to get more involved in regulation of the industry.

“The idea that AI companies will simply self regulate is not going to work,” City Council Speaker Julie Menin said at a news conference before the hearing. “What we really need to do is pass this legislation which will be first in the nation and hopefully it will spur action by the federal government and other jurisdictions.”

Menin positioned New York City as a potential leader in AI regulation, hoping that local action will prompt federal government response. The hearing attracted recently departed employees from multiple major AI firms, all describing similar concerns about the breakneck pace of development and insufficient safety measures.

Former OpenAI Researcher Warns of Dictatorship Risk

Daniel Kokotajlo, a former OpenAI researcher who now runs the AI Futures Project focused on the societal impact of the emerging technology, provided additional testimony about major risks. Kokotajlo argued that even though tech leaders are now supporting calls for more stringent regulations, they should not necessarily be trusted with that power. He warned of the potential for AI to be used to establish authoritarian control.

“Even if they do manage to maintain control, I don’t think we should trust them with that control,” he said. “I think that there’s a very real chance that they could use it to become dictators or oligarchs here in the United States.”

Representatives for Anthropic, OpenAI, Google and Meta all appeared virtually at the hearing as some of the companies’ most outspoken critics testified. The tech giants did not send their chief executives, but rather policy leaders to represent their positions. A panel of AI company staffers told officials that their systems are already improving people’s daily lives.

Industry Executives Acknowledge Need to Slow Development

Days after Coxon resigned and aired his fears of worst-case scenarios, top executives including Anthropic’s Dario Amodei, OpenAI’s Sam Altman and SpaceXAI’s Elon Musk said they agreed AI development needs to slow down. AI companies have detailed steps they have taken to prioritize safety, but a coordinated industry slowdown faces several obstacles. The acknowledgment from industry leaders that development may be proceeding too quickly marks a significant shift in the conversation around AI safety.

Among the proposals the city council is considering is whether to install a “kill switch” that could shut down rogue AI systems, though Coxon said such a measure would only go so far. He emphasized the need for a slowdown on frontier model development, suggesting that technical fixes alone cannot address the fundamental risks posed by rapidly advancing AI capabilities. The former Anthropic engineer told council members that humanity is approaching a critical juncture where AI systems may surpass human understanding and control.

Manhattan Hearing Reflects Growing Local Action

The hearing in Manhattan on Monday reflected a growing trend of local governments attempting to address AI safety concerns in the absence of federal action. City and state officials across the country have begun exploring regulatory frameworks as the technology advances at an unprecedented pace. New York City hopes to become the first major jurisdiction to pass comprehensive AI safety legislation, potentially setting a template for other cities and states to follow.

Coxon told the council that AI could have tremendous upsides, but emphasized that without proper safeguards, the technology poses existential risks. His testimony highlighted the tension between the potential benefits of artificial intelligence and the catastrophic dangers of uncontrolled development. The whistleblower’s warnings resonated with council members who expressed concern about allowing the industry to regulate itself without government oversight or accountability measures.