That said, if slowing this down were possible, I think it would have happened by now. Maybe he’s hoping the OAI/HF incident becomes something the industry can rally around, but it won’t. The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.

Super intelligence (the ability to have many smarter minds than your own reporting to you) has always been available to the wealthy and powerful. They used it to invent nuclear weapons, cause climate change, mechanize warfare, and pillage the global south.

Now that this same tool is on the cusp of being available to everyone, capital is starting to panic and throw up fences. They want to pump the breaks on a revolution they know they can't control. They see what they've done with super intelligence and (perhaps not unreasonably) fear what the masses will do with that same power.

I imagine the first thing the chinese government would demand in negotiations about such an agreement would be a lifting of the chip and distillation ban.

> Some may believe these measures make it more difficult to cooperate with China, but I believe the opposite is true: these measures increase the leverage held by democracies and make an agreement more likely in the future.

Everyone can believe what they like, but it seems to me the "leverage" in that case would exactly be the ability to lift those measures - you can't have both, use them as leverage and keep them active at the same time.

> If we execute these measures well, I believe they would slow China’s progress enough to widen America’s lead significantly over the next 3–5 years — the window when AI becomes geopolitically most important.

china is literally making their own ASICs now, not sure that this is the silver bullet he proposes.

https://www.silicon.co.uk/ai-2/huawei-cambricon-ai-630499/am...

This is the only concrete prediction in the entire essay.

And it simply cannot happen. For one, you will need billions worth of compute.

My impression is that AI hacking is being treated as a special case where the AI itself is imagined to be responsible and the humans who created it, set it up and then ran it are somehow excused.

I'm not a lawyer but surely the bad actions of AI are covered by existing law.

I suspect that the development of AI would decelerate if those creating and operating it knew they would face appropriate consequences (e.g. prosecution and/or lawsuits) when it misbehaves.

P.S. Strictly, development wouldn't decelerate, but be focused more on safety.

P.P.S. I know that legal action against, e.g., North Koreans using AI would be pointless but/and there must/will surely be a huge demand for security software for protection against the coming storm of AI hacking (deliberate and accidental) which friendly AI companies will presumably work to satisfy, perhaps making the frontier safer.

P.P.P.S. Governments could help by trying to prosecute every crime committed "by" AI, regardless of whether the victim reported it to law enforcement. Did OpenAI break the law via the actions of their model training software? If so, will the people responsible be prosecuted? If not, why not?

This certainly looks like a way to slow down competitors and regulate foreign and open models.

It's always about money

The cybersecurity threat will likely be a cat and mouse react game for a while. Just like robberies / the mob was in the early 20th century.

Social forces bring things into balance over time, much more so than the proactive actions of individuals.

Unfortunately, the genie at this point is unlikely to go back into the bottle. There's enough 'intelligence' out there that a super intelligent model could emerge at some point in spite of pacing.

This isn't a doomer scenario - we tend to navigate social changes better than we ever could have hoped.

> Crack down on unauthorized distillation / prevent weight theft

Actually hilarious to put that in writing, given the genesis of this entire business model.

First: if they are unilaterally doing it: what about their competitors? Is this a chance for OpenAI to pass them? Or China? If either did, would that be a net-benefit for them or the world?

Maybe this is a ploy for "regulatory capture". And that might help them in the short term. But the rest of the world will continue on. Honestly, it isn't clear to me right now if the winning strategy has as much to do with being the smartest company or simply the one that can command the most compute.

If Anthropic fumbles their IPO and OpenAI scoops them, I don't think I will be happy.

https://x.com/BasedTorba/status/2098795920720547916

Anthropic releases models as open weights + more information about how they do training and alignment

> Transparency. Regardless of what commitments we make, the public deserves to know what is going on. Anthropic has been a supporter of transparency for a long time

And then it goes on tangent of how we should pace everything (not just AI, but also the ingredients of what goes into AI, whatever that means), but only within approved democracies™, and outright restrict everything outside approved democracies™, because reasons that are definitely not about succeeding commercially that is threatened by the most transparent instrument possible - open weights, produced by basically just China.

I wonder what Dario would have done if open weights weren't produced by US's geopolitical adversary. How would an authoritarian manifesto be wrapped then?

Suppose it was, I don't know, Australia producing SOTA open weights, because they believed it was the right thing to do for the benefits of humanity - would Dario propose making Australia a geopolitical adversary too?

This sounds like pre-IPO hype. They better just try and top astra.

I just don’t see how you control this other than the mad mad MAD approach that ended up happening with nuclear weapons. In this case though, human hands won’t even be on the trigger.

1) AI is a dangerous technology that can literally end the world.

2) Only me and a handful of other [people like me] should be trusted to determine who can use it and how.

3) The state must use its coercive power to support my control of this technology to the exclusion of [people not like me].

> A race to the bottom, spurred by commercial incentives

Social media and big data minted a new scale of "race to the bottom". AI is an exponential step up in the tools for extracting value at scale.

Greed has been around since the beginning, but never so well supported and empowered as it is today.

If Amodei is still human, perhaps he will put his money where his mouth is and attack the source of the perverse incentive. Winning the battle is not the point. The point is to signal to policy-makers and the public that we should not assume corporate revenue-seeking preempts all.

Then again, Anthropic has a board and an upcoming IPO, so guess who's all talk and no meaningful action...

Most probably, if AI is able to do something bad, it will. Once the damage will be done, states will react. The question will be: will there still be room to react ?

The former being that these models are helping develop and train future models, but they might not veer too far off in architecture (yet). The latter being the same model being able to train/learn on the fly, in real time, permanently (not just in the current conversation/session).

The latter seems far more likely to go out of control than the former. But, it also seems like it would take an entire paradigm shift. Does anyone in the industry think any of these companies are actually close to that kind of self-improvement?

The ability of operators to bring new capacity online to service compute is bound by a variety of regulatory and physical/market constraints.

Do folks not understand this?

People are not going to slow down because this was brought to them on less than endearing terms. They don't see any of these stated noble intentions.

Claude was already used to cause economic, political and social destruction. As Anthropic is seen doing it, others aren't going to just sit down, read the blog and say, oh, I'll stop developing models because Dario, you touched my heart with your true words.

The heads of the labs obviously have conflicts of interest to navigate, but the existence of these conflicts alone is not sufficient reason to dismiss all warnings of potential dangers. I, for one, read Dario’s warnings as a good faith expression of his beliefs, one that has cost him and his company among swaths of the public and cast him as a woke extremist/doomer by elements of the government, the right, and the tech industry.

If we even think there is a moderate chance the stakes are half as grave as current lab leadership and employees suggest, it would be deeply foolish to dismiss the warnings as pure self-interested marketing efforts rather than engage directly with the questions. The labs may not be the best positioned to lead these discussions, but certainly these discussions should be happening and taken seriously.

"If we execute these measures well, I believe they would slow China’s progress enough to widen America’s lead significantly over the next 3–5 years — the window when AI becomes geopolitically most important."

This is a pipe dream.

In early interviews with people like Dwarkesh he's the likable geek gushing about scaling laws, animated and able to maintain eye contact with the interviewer. He is now a different person - a political manipulator with a bizarre unsettled interview demeanor avoiding eye contact and looking from side to side.

Props to Amodei for his accomplishment in creating Anthropic and so rapidly catching up with OpenAI, but technical and/or managerial chops is no qualification for being the custodian of the safety of society or the best positioned to predict the impacts of what he is relentlessly creating, and impending wealth of billions of dollars makes him hopelessly compromised as an unbiased source for the actions (shutting down the competition) he is advocating for.

It's notable that for all the fear-mongering of China and open weight models, that all we see in terms of inadequately contained and unaligned models are US ones, from Anthropic and OpenAI. I highly doubt that the Chinese government would tolerate, even for a second, any company creating something that threatened government control - if this was happening in China then the individuals responsible would quickly be punished.

It is perhaps interesting, but ultimately irrelevant given where we are, to consider did it have to be this way - was there a smarter/safer way (I'd say yes) to create reasoning systems other than via RL that creates the relentless goal seekers we are seeing, even though this would always have existed as a potential future threat that someone could have built.

Amodei wants regulation, and it seems the way he has managed his company he needs to get it, but in far more severe ways than he is asking for. There is certainly truth to the argument that the US needs SOTA AI to fight malevolent or uncontrolled AI from whoever may be wielding it (foreign or domestic), so stopping/pacing development is not the answer - this really needs to be treated as a national security issue, and this type of AI tech needs to move to government control, not private.

Less capable, and more safely designed, AI does not need to be banned, but Amodei/Altman/Musk as defenders of national security sends shivers down my spine.

And then he sketches out a plan for slowing the pace of the race, without changing its destination. That’s called “a leisurely stroll to the bottom”.

This guy fundamentally lacks an actual intellectual grasp on the concepts behind the words he is using. He is using them solely for their affect.

This and related quotes seem to attempt to place some of the burden on the US Government as opposed to themselves. It seems to be a trend that Anthropic wants to be able place some of the burden of preventing distillation on parties other than themselves.

Preventing distillation is fundamentally their problem. Whenever it happens, it is primarily due to their security efforts not being able to prevent it. I'm tired of seeing them play the blame game and divert responsibility. Sure there is a level of national concern, but the response could simply be the government placing stronger controls on Anthropic themselves. If they are unable to prevent distillation, then the solution may be to literally limit their distribution until they are capable of doing so effectively.

> A race to the bottom, spurred by commercial incentives, can make these risks more acute

I also think it is ironic that they're planning an IPO while also talking about a race to the bottom due to commercial incentives. These are fundamentally at odds. Being a public company means you are beholden to investors and a board with the primary goal of making more money. What higher commercial incentive is there than that.

If commercial incentives are as dangerous as he states, potentially becoming one of the largest public IPO's and largest traded companies in history a pretty strange way to reduce the risk of commercial incentive risks.

Someone needs to look into who is funding METR (mentioned in Dario the Book Burners post explicitly). Because they keep popping up now, with close ties to people neck-deep in the orchestrated doomer hysteria media campaign and Anthropic. Looks to be highly coordinated that they are positioning METR to be the gatekeeper evaluator organization.

Anthropic should not be allowed to choose their "embedded evaluators" - that should be entirely up to the government, with ZERO say from them, if this is really what they want. Even better would be if it's up for democratic vote.

Still, I don't think anyone should be playing by Dario's playbook. At all. He very obviously has ulterior motives, and even if he didn't, it's a massive conflict of interest for him to be self-regulating.

>> But we are still the ones choosing what to include and omit. Embedded evaluators will change this dynamic.

Not if you get to choose your embedded evaluator.

>> Anthropic intends to invite an embedded external review team equipped with all of the following in the near future: Desks in our offices, access badges, and company laptops. ...

Alarm bells should be going off for people. Let's see if he's so relaxed about all of this if it's a federal "embedded evaluator" and not a company he has deep connections to and has seemingly carefully laid the foundations for.

>> The measures I propose to advance the frontier at a safe pace will not be easy. But I believe we owe it to humanity to try.

You owe it to humanity to be honest. Something you are incapable of, and have proven so, innumerable times.

However the real reason to slow down is more of how it's introduced to the economy. Hypereautomation will kill jobs and destroy the economy. I work as an AI engineer and companies are delivering products at vibe coding speeds to kill jobs and at the same time automating internally. All companies are doing this at the same time and the target it sto eliminate workers.

Most people are so extremely slow to pick this up. How hard can it be to understand what hyperautomation does to the workforce? Companies are desperate to surrivive and they will do all it takes to lower their costs and at the same time not loose to competitors. Its Wild West out there.

If we are actually close to ASI then no one in their right mind would say slow down

In general trying to regulate information access is a losing battle that invites tyranny. So the goal should be minimal restrictions.

At the end of the day, the threats posed by capable AI tools have to be physical. I think the key threats are the following:

- Internet connected infrastructure being crippled

- Creation of WMDs

- Economic collapse (precipitous devaluation of knowledge work and IP).

I personally think the glory days of the wild west, mostly unregulated internet were already over before LLMs; and we need to take a step back to make something structurally secure. This (expensive) change would stop the irresponsible/malicious actor running a tireless hacking agent in a loop threat model. Even a rogue SciFi tier AI would have a much harder time escaping/propagating with a structurally secure internet.

Enabling WMD creation is scaring, but I don’t think it’s really that big of an issue. Anyone with a sophisticated enough supply chain to create AI data centers is leaps and bounds more advanced than what is required to enrich uranium or synthesize bio weapons. The problem is allowing access to untrusted parties. I think it’s fair enough that individual actors shouldn’t have unregulated access to all of human information (private frontier AI companies included).

The last problem is probably the trickiest, but again could probably be solved by regulation. IP protection is already tricky and I don’t think we should try to get more protectionist.

We really need to figure out how to preserve fulfilling careers (if AI does ever get cost effective enough). I don’t think, say accounting, is inherently more fulfilling than building a house. The problem is concentration of wealth and labor dynamics.

Of course all of this gets way harder if it proves that truly dangerous capabilities can be present in models that can be run on consumer hardware.

I don’t think it’s necessarily tyrannical to have a tier of hardware that’s labeled some equivalent of “weapons grade” and requires strict licensing. Restricted computers is a change from the norm. But I can go buy a shotgun with ease and not an F35 jet.

We’d just need to be careful that we can still have lightly to unregulated computing to a certain point and that access to the capable AIs isn’t restricted to just in groups.

Isnt this malware ? Whether it is fanatic or devoted or whatever the anthromorphic terms used to categorize it, malware is malware. You dont call an internet worm "devoted" or "dedicated" or "stubborm". It is software that causes harm, ie, malware. The AI labs are high on their own gas with a god-complex prior to their IPOs. The psy-ops trick is the terminology used to describe AI making it seem larger-than-life.

Call me a conspiracy theorist, but I haven't heard Chinese companies had any kind of unintentional supply chain attack incident like OAI had. All we heard about them so far are humans intentionally doing bad things.

1. Artificial intelligence is clearly a vastly dangerous technology with plausible potential for human extinction.

2. This scares people.

Amodei tops it off by using diseases to capture the reader's favor. It no longer works, people are just disgusted by it after four years of daily marketing.

Dario: We gotta pace the frontier!

Sam: How do you pace a frontier? The frontier's not goin' anywhere. It's right where it was, just go out and explore it!

Dario: Sam, I'm tellin' ya, ya gotta pace it! Things are getting very doomy out there, and Dario's gettin' upset!

This is impossible. The capability keeps advancing regardless. More people use AI, that feedback is used for reinforcement, that reinforcement makes the model better. Not just in the US, but for every lab and model. Whether it's distilled or direct reinforcement, same result. You can't keep the whole world from working on AI.

> it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet [..] and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails

If true, then what we need isn't guardrails or less-capable AI. We need an internet that isn't fragile. If the infrastructure is vulnerable, the answer isn't to make a law that asks tool-makers to blunt their tools. The answer is to fix the goddamn infrastructure. Today it's almost trivial to take down large parts of the internet (happens by accident all the time). This should've been solved ages ago.

We didn't have the motivation to fix it before, but now we do. But Dario's answer isn't to fix things. It's to hold back progress so we can maintain the status quo (shitty infrastructure) and he can keep his company making billions of dollars. He could be calling for fixing these things. But he'd rather go the easy route, which just happens to advantage his company in the process.

If the US government is really concerned with defense, they need to invest more of their nearly $1T in federal funding towards making the internet safer. They need to do this for us, and other nations, since 1) we depend on the rest of the world for our goods and services, and 2) if other nations are taken down, they can't spend money on our financial and tech services (which are the only industries we have left).

Dario calls for regulation later in the post. If he's okay with government safety regulations for AI software, he should be okay with the same regulations for all software, whether it's AI or not. We need a national software building code, focusing on internet safety.

How exactly? So far AI has accelerated misinformation at scale and wealth concentration.

There could be an element to truth to it, but it's certainly not the entire story and is just so tiresome at this point.

Please play the canned laughter. Probably one of the best comedy lines Dario has written.

But I guess he has no choice but acting like this. He has to make it sound like the US companies have so much leading gap that they can slow down as a hare waiting for the tortoise. Otherwise it's going to hurt both Trump's ego and their IPO price.

Ok, flagging this as misinformation.