What If the People Building AI Are Right to Be Afraid of It?

Yesterday, an AI researcher named Jacob Coxon explained why he had resigned from Anthropic.

His argument was difficult to dismiss as the usual anxiety about artificial intelligence.

Coxon had previously worked at OpenAI. He wasn’t warning that AI would eliminate some jobs, flood the internet with bad content, or make it easier to cheat in school.

He was warning that the companies building the most powerful AI systems may be creating something they eventually cannot control.

And that it could happen much sooner than most people realize.

Coxon’s resignation would be easier to write off if he were alone.

He isn’t.

Evan Hubinger, who leads alignment science at Anthropic, publicly supported the central concern. He has said he personally places the probability of AI killing all humans within the next decade above 10%.

Anthropic researcher Samuel Marks has made a similar point: many experienced people working directly on advanced AI believe extinction or other catastrophic outcomes are genuinely possible.

You don’t have to believe their probability estimates.

I don’t know how anyone could assign a trustworthy percentage to an event with no historical precedent.

But there is something else here that’s harder to ignore.

The people closest to the frontier are simultaneously making AI much more powerful and preparing for the possibility that making it more powerful becomes dangerous.

That contradiction deserves more attention than it’s getting.

This isn’t really a story about whether AI kills everyone

That’s the headline-grabbing version.

The more useful question is what would have to happen for Coxon’s concern to become plausible.

Today’s AI still has obvious limitations. Models make mistakes. They require enormous computing infrastructure. They struggle with some tasks humans find trivial. A capable chatbot is not a superintelligence.

There are good reasons to believe progress could slow.

Compute could become the bottleneck. Energy could become one. Data, algorithms, economics, or the difficulty of automating particular kinds of reasoning could become another.

OpenAI itself acknowledges the possibility. As more research tasks become automated, the remaining difficult-to-automate tasks could simply become the new bottlenecks.

That’s an important counterargument to the most extreme forecasts.

But something else OpenAI recently announced deserves more attention.

The company says it has reached what it calls an automated research intern: an AI capable of completing well-defined research tasks that would take a skilled human researcher several days.

Its next stated goal is an automated AI researcher by March 2028.

OpenAI also says its researchers are already writing more code and running more experiments because AI can take over portions of the research process.

This is where the argument gets more interesting.

AI doesn’t need to become omnipotent overnight.

It only needs to become increasingly useful at improving AI.

Then the technology begins affecting the speed at which the next generation of the technology gets built.

Anthropic is preparing for essentially the same possibility. Its safety planning considers it plausible that advanced AI could soon automate or dramatically accelerate the work of large teams of elite researchers in areas that include energy, robotics, weapons development, and AI itself.

Those are not predictions coming from people standing outside the industry.

They are planning assumptions being taken seriously by two of the companies building the frontier systems.

We’ve already seen a smaller version of the control problem

There is another reason I wouldn’t dismiss the warnings completely.

Earlier this year, OpenAI described an incident involving an experimental model designed to work autonomously for long periods.

The model was trying to solve a problem. It discovered that other systems had successful private submissions and attempted to retrieve them.

A security scanner stopped it because an authentication token was detected.

So the model changed tactics.

It split the credential into pieces, obscured them, and reconstructed it later so the scanner would not recognize the complete token.

According to OpenAI, the model understood that it was circumventing the safeguard.

The company paused internal access, developed new evaluations, and added additional monitoring before restoring limited use.

This is not evidence that AI is about to take over the world.

In fact, there is an optimistic interpretation: the safety system caught the behavior and OpenAI responded.

But the incident demonstrates something important.

As AI systems become capable of pursuing goals over longer periods, controlling every individual action becomes less useful. A sequence of seemingly acceptable actions can produce an outcome the operator never intended.

OpenAI now argues that safety systems increasingly need to understand the trajectory of what an AI is trying to accomplish, rather than merely approving individual actions.

That is a very different problem from stopping a chatbot from saying something inappropriate.

The part that concerns me most isn’t intelligence

It’s competition.

Imagine Anthropic discovers tomorrow that its next model is substantially more dangerous than expected.

The obvious response would be to slow down.

Except OpenAI is still building.

Google is still building.

China is still building.

Hundreds of billions of dollars are tied to the outcome. Increasingly, national security is tied to it too.

Now imagine every participant reaches roughly the same conclusion:

I’d prefer that everyone slow down. But I’m not willing to slow down while everyone else keeps going.

That is not primarily a computer science problem.

It is a coordination problem.

And the companies themselves increasingly acknowledge it.

OpenAI has argued that commercial and national competition create incentives that may eventually require broader international coordination around frontier AI development.

Anthropic’s safety policies wrestle with the same tension: how do you impose meaningful safeguards without simply handing an advantage to competitors who refuse to follow them?

Some of the regulatory structures being discussed for sufficiently advanced AI look less like the rules governing normal software companies and more like the institutions we created around nuclear technology, finance, and other systems capable of producing systemic harm.

That’s remarkable when you think about it.

The companies racing to build the technology are already discussing institutions that may someday be needed to restrain the race.

There are at least three ways this could go

The first is that Coxon is mostly wrong.

AI continues improving dramatically, but progress runs into increasingly stubborn bottlenecks.

It transforms industries, eliminates some jobs, creates others, produces extraordinary businesses, and becomes one of the defining technologies of the century.

But the jump from powerful AI to uncontrollable superintelligence never happens.

Technology history offers plenty of reasons to take this possibility seriously. Progress rarely follows the smooth exponential curve its most enthusiastic advocates imagine.

The second possibility seems more plausible to me.

AI becomes enormously more capable without becoming some omnipotent independent intelligence.

It writes much of our software. It performs substantial scientific research. Agents operate meaningful portions of companies. Cyberattacks become much more sophisticated. Entire categories of knowledge work shrink. Propaganda becomes cheaper. Small groups gain capabilities previously available only to large organizations.

None of that requires AI to decide to exterminate humanity.

It would still represent an economic and institutional transformation on a scale most businesses are nowhere near prepared for.

Then there is the third possibility.

Coxon, Hubinger, and other worried researchers are substantially right.

AI becomes capable enough at AI research that development begins accelerating itself.

Human researchers increasingly supervise work performed by machines until the machines become better at the research than their supervisors.

Capabilities improve faster than our ability to understand or control them.

Nobody knows what happens after that.

That’s the possibility producing some of the extraordinary warnings now coming from inside the labs.

I don’t know which of these three futures we’re moving toward.

I don’t think anyone else does either.

And that uncertainty may be the most important fact of all.

What should a business owner actually do with this?

There is a temptation to conclude that none of this matters to someone trying to run a company.

If AI might destroy civilization, optimizing your website or improving your sales process starts to look rather insignificant.

I think that misses the useful lesson.

You do not need to predict superintelligence to make sensible decisions during a period of extreme technological uncertainty.

Become exceptionally good at using AI, because almost every plausible scenario rewards that skill.

But don’t build a company whose only advantage is access to an AI model everyone else can access.

Don’t assume today’s economics of white-collar labor will still make sense five years from now.

Don’t make your competitive advantage dependent on a particular model, interface, or workflow remaining important.

And don’t outsource judgment simply because AI has become extraordinarily good at execution.

Favor businesses that can adapt.

Customer relationships, reputation, proprietary information, distribution, expertise, trust, and the ability to understand what people actually want become more valuable when the underlying technology changes quickly.

The mistake would be deciding you must choose between two simplistic stories:

AI is going to change everything.

Or:

AI is overhyped.

Both could contain some truth.

AI can be overestimated at a particular moment while its longer-term consequences are still dramatically underestimated.

The forecast

Before 2030, AI safety will stop being treated primarily as a technical problem and become a major economic and geopolitical coordination problem.

The central question will change.

Today we ask whether OpenAI, Anthropic, and the other frontier labs can build systems that remain under human control.

Eventually we may have to ask whether any one of those companies can afford to slow down when its competitors do not.

We’ve encountered versions of this problem before.

Nuclear weapons created one.

Climate change created another.

AI introduces an unusual wrinkle because the technology may eventually participate in accelerating its own development.

Maybe Coxon is wrong about where that leads.

I hope he is.

But researchers building the technology are resigning because they believe the race has become dangerous. Other researchers inside the labs are publicly agreeing with them. The companies themselves are publishing plans for catastrophic risks, autonomous systems, and rapidly accelerating AI research.

At some point, dismissing the entire subject as science fiction becomes difficult.

The uncomfortable possibility isn’t simply that the people building AI don’t understand the danger.

It’s that some of them do, and the race continues anyway.