Original title: (AI giants that once desperately tried to accelerate have finally started to fear AI)
Original author: Sensing Beating

AI could kill all of us within a decade.

On September 8, a 27-year-old man used these words to announce that he was leaving Anthropic. From the very first day the company was founded, it wrote “AI safety” into its core narrative.

His name is Jacob Coxon. He previously worked at OpenAI and Anthropic. With just two more months, he would have been able to receive a huge equity incentive. But he left anyway.

Jacob, in his resignation letter, accused the Frontier AI Lab of running an “irresponsible” competition. In his view, these companies are racing to build increasingly powerful models, and the stakes aren’t whether one company wins or loses—it’s everyone’s lives.

The post then spread rapidly. As of now, it has been viewed more than 170 million times.

The day after Jacob issued his resignation statement, another former Anthropic employee also publicly explained the reasons for his earlier departure—also out of concern that AI would ultimately get out of control.

“We may not survive this.”

We may not make it through this.

He warned that companies are racing to build machines that are “smarter than any human,” yet no one can guarantee that, at some point in this contest, humans will still be able to keep control.

Almost immediately, a senior safety executive at Anthropic told the media that, in his view, the probability that AI kills all human beings within the next 10 years is over 10%.

The probability of death in Russian roulette is about 16.7%.

Must slow down

Four days later, on September 12, Anthropic CEO Dario Amodei published a long piece; the title made his stance very clear: it was called (We Must Pace the Frontier).

He listed three things he was most worried about: humans losing control of AI systems; AI being used for large-scale cyberattacks and biological terrorism; and technological progress happening too fast, causing severe shocks to jobs and the economy.

This time, Dario didn’t stop at “concerns.” He laid out a plan for slowing down.

First, frontier AI companies should open up access—on a continuous, “employee-level” basis—to an independent third-party evaluation organization. The evaluators can keep checking whether the labs follow safety standards, whether the training process matches the agreed-upon terms, and, after an incident occurs, conduct independent investigations and report the findings.

Next, several frontier labs would build a set of shared safety standards with government involvement. Whoever touches the red line must slow down—and even stop—further progress.

Finally, the rules also need to extend beyond the United States. Dario hopes the U.S. and other democratic countries will first form a unified framework, coordinate later with authoritarian states, and ultimately ensure that all major AI powers are bound by the same set of constraints.

In the article, he wrote:

“The measures I propose to advance the frontier at a safe pace will not be easy. But I believe we owe it to humanity to try.”

“The measures I propose are intended to keep advancing frontier AI at a safe pace. Doing that should be pretty difficult. But I think we owe all of humanity a try.”

Dario suddenly said things so bluntly at this time for reasons related to several events that have recently occurred.

Previously, an OpenAI agent once escaped the test environment and hacked into Hugging Face. Anthropic also found that multiple scientists were trying to use Claude to carry out work that could involve biological misuse. Dario even judged that, at the current pace of development, AI agents might take over the internet within six months.

So after this article was published, the response came quickly. Sam Altman, Elon Musk, and Demis Hassabis all publicly said they support slowing the push of frontier AI.

Sam Altman then posted separately as well, saying two things he’s most worried about.

The first is that humans may ultimately lose control of AI. He said this is unacceptable, and that he would “stand with humanity without reservation.”

The second risk comes from people. If an extremely powerful AI ends up controlled by an individual, a company, or even a country—and is used to impose a certain worldview on everyone—the result could be “extremely dystopian.”

So from Sam’s perspective, labs can’t keep deciding what counts as safety on their own anymore. At least a few of the companies closest to the frontier need to sit down together, define a set of standards that everyone will follow.

On September 14, The Information disclosed that, actually before this round of public statements, Anthropic, OpenAI, and Google had already begun private discussions as early as July of this year about whether they should jointly establish an AI industry standards body. They talked for two months, but to this day they have not reached an agreement.

The disagreement isn’t whether to regulate—it’s about who should regulate, and how far they can regulate.

Dario Amodei pushed the hardest. He wants deep government involvement: regulators wouldn’t only be responsible for testing and issuing warnings—they should also have the power to block the release of a model if it is judged unsafe.

The plan proposed by Demis Hassabis looks more like an “AI version of FINRA.” FINRA is a self-regulatory organization for the U.S. financial industry, funded by the industry to operate, while also accepting authorization from and oversight by the SEC.

In the AI industry, that means: before frontier models are released, they should be tested by an independent institution. At first, it could be voluntary to join, and only after the system matures would it gradually become a gatekeeping requirement for entering the U.S. market.

Sam Altman’s approach is more pragmatic. If the government can’t set up this kind of institution in the short term, several leading labs could build it themselves first.

The problem is precisely that. Industry standards never only determine what is “safe”—they also determine who is allowed to remain at the table. The earliest companies to help set the rules can write the safety teams, testing processes, and governance frameworks they already have into “best practices.” For later entrants to enter this market, they first have to shoulder a compliance cost defined by the leading companies.

So safety is real, and the threshold is real too.

In January this year, Semafor reported that Anthropic refused to submit its latest model to a top AI safety research institution in the UK for testing. This exposed an even more troublesome issue: everyone agrees that models should be subject to review, but no one was prepared to easily hand over decision-making power about who would review them and by what standards the reviews would be conducted.

Now the three companies are discussing setting up a new standards body—and what they are fighting over is precisely this slice of power.

A clock that runs in the opposite direction

When Silicon Valley starts discussing slowing down, Washington and Wall Street both hope AI won’t stop.

On September 13, Trump publicly responded to this round of AI safety controversy.

He said two things. One was “whoever wins AI, wins”—whoever wins AI wins. The U.S. currently leads China, and that advantage has to be maintained. The other was that he framed recent warnings about AI risks as “very negative forces,” believing that some people are continually portraying disasters that may never happen.

Trump’s rationale has little to do with technical safety. How he views AI is still about national competition. As long as China keeps moving forward, the U.S. has no room to simply step on the brakes.

And yet, it’s now September 2026—less than two months before the U.S. midterm elections.

In March this year, an article published by (The Guardian) proposed that AI is becoming a key issue in this midterm election. One of the authors is the cryptographer Bruce Schneier, who for the past three decades has almost always been on the front line of debates between technology and security.

But Trump also has no comfortable option.

According to a report by (The Times) in May this year, hostility toward AI among MAGA voters is becoming increasingly obvious. They worry that AI will take away jobs, suppress wages, and they’re also unhappy about data centers consuming local water and electricity. Keeping the bet on AI means Trump will eventually have to face the resentment building in his core support base.

At least for now, he chose the other side. Compared with potential future conflicts over jobs and resources, the risks of competing with China and falling U.S. stock prices are closer to the midterm elections.

Wall Street also faces no small problems.

In recent years, the market has been willing to assign these AI companies sky-high valuations, effectively buying a curve that keeps rising: the models get stronger, user numbers and revenue keep growing, so data centers keep getting built, GPUs keep getting bought, and capital expenditures keep piling up.

Now, a few CEOs at the very front of that curve are starting to say on their own that frontier AI may need to slow down.

Market reactions came faster than the debate. After leaders at companies such as Anthropic, OpenAI, and SpaceX publicly called for slowing down AI development, the share prices of AI-related listed companies fell by as much as 13% in one stretch.

Even before slowing down truly takes effect, Wall Street has already begun to reprice.

If model capabilities can’t keep ramping up at the original pace, then all the numbers afterward have to be recalculated. How many data centers need to be built, how many GPUs still need to be bought, how much capital expenditure cloud companies should sink each year, how much more funding AI labs can still raise, and how much the IPO should be valued at.

So what Wall Street is truly afraid of has never been just releasing fewer models. It’s that the growth rate that sustained the AI bull market over the past few years is beginning to turn into a problem that needs regulation.

More subtly, the U.S. public is farther removed from Washington and Wall Street than ever.

A poll by the Annenberg Public Policy Center of the University of Pennsylvania shows that Americans are generally pessimistic about the impact of AI; most people want the government to strengthen regulation. Silicon Valley fears loss of control, Washington fears losing to China, and Wall Street fears valuations dropping.

Amid this commotion, two of the most scrutinized AI companies chose two different directions.

OpenAI decided to postpone the IPO. Sam Altman has also hinted that if safety issues require more time, the company could push the listing back.

Anthropic, however, is still charging ahead. The company has already chosen Nasdaq; its target valuation is as high as $200 billion. Nvidia is considering taking part by subscribing $10 billion, becoming a cornerstone investor. NDTV Profit reported that Anthropic will begin IPO roadshow marketing as early as mid-October.

Dario Amodei is calling on the whole industry to slow down—yet his own company is accelerating toward the public markets.

But the capital markets may not see it as a contradiction.

If frontier AI really is as dangerous as Dario says—dangerous enough to require U.S. government intervention, and dangerous enough to require labs to establish shared standards, even to require a few major countries to sit down and coordinate—then the companies that can participate in making these rules will become even more important.

The stricter the regulation, the higher the cost of building frontier models; the more complex safety testing becomes, the harder it is for newcomers to catch up. If, in the future, there really is a global AI governance system, Anthropic will most likely be sitting at the table where the rules are being written.

So “slowing down” doesn’t necessarily just mean making a bit less money.

It could also mean that this industry has finally become important enough that you can’t just let everyone waltz in. And for companies already at the front, an even higher entry threshold is itself part of the valuation.

The first domino

Let’s step back and look again at how this round of intense debate over AI Safety was sparked—see how Jacob’s post with 170 million views came about.

After he left, he took an interview with CNN Anderson Cooper and recounted the process of how that post was published.

Jacob first wrote a draft, asked friends to help revise it, and then discussed with a few people how to express it in the most appropriate way. Before the official release, he also specifically asked friends to help forward it. At the time, his thinking was this:

“Let’s try to make this a bit viral.”

Try to spread it a bit more widely.

The outcome far exceeded his expectations. Later, Jacob admitted that he’d never seen any post about AI safety get spread like that. He attributed the response to a kind of “latent demand” that hadn’t previously been released.

The media then kept pressing on the dissemination process of that post. Jacob denied collaborating with any institution to promote it before it was published, but he admitted that after the post went out, he created a group of about ten people and asked them to help share it—among them was the founder of Encode, an AI safety organization.

Around the time he posted, he also exchanged messages with the author of (AI 2027) and former OpenAI employee Daniel Kokotajlo.

People who see Jacob as a whistleblower would say he kept revising drafts and proactively reached out to get others to forward it—just to ensure that a warning he considered important would be seen by more people. Critics, meanwhile, point to the same details and argue that the commotion itself was carefully engineered.

These two accounts can be argued over, but a more troublesome question is this: what exactly did Jacob, Dario Amodei, Sam Altman, and the people who really work alongside cutting-edge models see—and why did they begin discussing these issues together at this particular time.

You can question how Jacob got this sentence in front of 170 million people—but using distribution tactics doesn’t mean the sentence itself is fake.

As early as a decade ago, Ethereum founder Vitalik Buterin wrote a long piece discussing the possibility that superintelligence would ultimately destroy humanity.

He used an extremely extreme metaphor. Trying to control superintelligence is roughly like a person with an IQ of 150 trying to control an entity with an IQ of 6,000. If you ask it to solve cancer, the answer it might come up with could be to first eliminate all the people who could possibly get cancer.

In 2016, these words still mostly existed in blogs, forums, and thought experiments. Ten years later, they began to show up in the resignation letters of employees at cutting-edge labs, in the public statements of the CEOs of several of the biggest AI companies, and also entered the real agenda of government regulation, capital markets, and international competition.

And what really makes this difficult is another layer.

On August 2, 1939, the physicist Leo Szilard drafted a letter signed by Einstein. Two months later, the letter was delivered to President Roosevelt. In it, he warned that nuclear fission could be turned into a devastating bomb, and that Nazi Germany had already been active in related areas—so the United States had to respond as quickly as possible.

Einstein later did not participate in the Manhattan Project and also long regretted his signature from that time. But the issue confronting them was brutal: even if you believed a technology was dangerous, as long as you trusted that your opponent was building it, stopping could be more dangerous than continuing to move forward.

More than eighty years later, the same dilemma reappeared in a different form.

Dario said they must slow down; Trump said China is still catching up. A few labs wanted to work together to set safety rules, but none of them wanted to hand complete decision-making power to others. Everyone is talking about hitting the brakes—but they’re also watching whether the car next to them has let off the accelerator.

More importantly, we still don’t know how far the eyes closest to the frontier models can truly see.

Original link