Anthropic S-1 filings trigger debate on existential risk and 'Pacing the Frontier'
Policy experts Nathan Calvin and Dave Banerjee highlight that Anthropic's S-1 risk factors mention catastrophic and existential risks with passive wording. This has drawn criticism alongside debate over the company's 'pace the frontier' strategy, which some view as a commitment not to slow development unilaterally.
All posts · 30, most important first
- 3 Mentions of catastrophic + existential risks in Ant S1: • Investors may be hurt because Anthropic models may behave in unintended/catastrophic ways • Anthropic may prioritize safety in a way that hurts investors • Anthropic may receive bad media coverage about risks
IT'S OUT: Anthropic's IPO S1 sec.gov/Archives/edgar… open on X7 likes · 6 reposts - This part of Anthropic's S1 on monitoring/alignment risks is written in a weirdly passive way: "If we become less able to detect or assess misalignment as our models advance, we may deploy models whose behavior we cannot adequately verify" Somewhat by definition, shouldn't decisions around deployment be predicated on being able to "adequately verify" model behavior? E.g. if Anthropic can't adequately verify model behavior, they could also just choose not to deploy the model. (though of course "adequate" may be in the eye of the beholder)
All time risk S-1 risk factor from $ANT open on X22 likes · 2 reposts - Great catch by Matt Levine in Anthropic's S1: "Concerns have been raised... within our organization... that sufficiently advanced AI systems could... pose catastrophic or existential risks to humanity" "Concerns have been raised" is a very strange and passive way of saying that ~all of Anthropic leadership have been warning about these risks for yearsmatt predictably sharp on ANT S1 but goddamn is he always funny too bloomberg.com/opinion/newsle… open on X23 likes · 0 reposts
- Anthropic says that even the entire insurance industry in the aggregate will not have the capacity to insure the catastrophic risk that ANTHROPIC ITSELF poses to the world lol
13 likes · 4 reposts - Anthropic claims that their "call to pace the frontier... does not describe [their] own model development or release timelines" wut?? The whole point of pacing the frontier is to slow down AI progress, which NECESSARILY slows model development timelines... sus
2 likes · 0 reposts - "The justification for racing rests on Anthropic fighting for strong governance, and Anthropic's record on regulation points the other way. Together with its breakneck racing, that fits a much darker picture of the company than the one that seems to prevail among your employees. I hope that Anthropic reflects on this picture, supports stronger regulation, exercises extraordinary caution when advocating for weak regulation, and doubles down on its founding purpose of helping the world safely navigate transformative AI together, instead of prioritizing Anthropic’s ability to win a race that imposes intolerable risks on everyone alive."@_nathancalvin: Dario — Thank you for taking the time to respond to my previous post. I think it's great when this kind of engagement happens in public rather than on company Slacks, and I hope to see more of it. I agree that Anthropic isn't best modeled as a profit-maximizer. Your post gave several compelling examples of Anthropic taking competitive hits for safety, and insofar as my previous post suggested otherwise, that's on me. But I don't think I made my main point as clearly as I could have, so let me try again. It comes down to four key claims: 1. Anthropic seems to be racing at breakneck speed to build recursively self-improving superintelligence (RSI). By its own lights, that could kill billions of people or literally destroy the world. 2. Risking destroying the world is very hard to justify. It is only conceivably acceptable if serious attempts to put RSI under robust societal governance are failing. 3. Far from meeting that burden, Anthropic, when push comes to shove, often works to keep governments from tying its hands. It seems to care much more about preventing government overreach than about reducing the risk that it and other AI companies will destroy the world. 4. Even if the AEGIS Act really was an honest mix-up, as you suggest, that is itself a serious problem. It suggests Anthropic isn't treating the governance of this technology with anything close to the gravity it deserves. In greater detail for those who want it: 1. Anthropic is racing to RSI, and by its own lights that is insanely risky. Launching RSI could literally destroy the world. That's according to you, your co-founders, and most of your safety staff and capabilities researchers, not to mention many AI researchers outside your walls. You are intentionally trying to create generally intelligent systems more capable than all humans that you claim in six to twelve months “could be capable” of taking over the entire internet before we have the ability to reliably monitor, control, or align them. “Succeeding” at RSI could kill or permanently disempower billions of people in the near future, including me and everyone I love. Yet Anthropic’s primary focus seems to be marshalling an immense amount of resources and talent in a race to get there first. 2. Risking destroying the world carries an enormous burden of justification. I believe the vast majority of people do not want Anthropic to do this (to the extent they know what it's doing). Imposing even a modest chance of catastrophe on billions of people who never consented requires an extraordinary justification. The central one Anthropic has offered: someone will build RSI regardless, so it's better for a safety-focused lab to be at the front of the pack, using that position to support strong AI governance. (Other reasons Anthropic has given, like modestly accelerating improvements in healthcare or economic growth, don’t seem remotely proportionate to the risks Anthropic is imposing by its own lights.) But that argument only works as a last resort. Racing to RSI is only conceivably okay if attempts to institute robust societal governance to manage that race are failing despite your best efforts. A lab that relies on this justification should be a strong, consistent supporter of efforts to enact governance that could bind it, including governance that could slow it and significantly constrain what it does. 3. Rather than supporting efforts at strong regulation, Anthropic has intervened to keep governments from tying its hands. Anthropic has sometimes helped: it played a very helpful role in passing transparency bills in Illinois and ongoing efforts in Massachusetts, and I'm sincerely appreciative. But transparency is far from enough for a technology that will be the world's most powerful superweapon at best, and an uncontrollable swarm of lethally superintelligent agents at worst. When it matters most, Anthropic has often pushed for weak regulation. The AEGIS Act is the most obvious example. No highly risky industry is regulated as anemically as AEGIS would regulate frontier AI: not nuclear, not banking, not aviation, not food safety, not the manufacture of explosives. In all of those industries, specialist regulators can address unsafe practices well before they cause harm. AEGIS would bar the federal government from intervening to block unsafe development or deployment of models until there is an "imminent risk" of catastrophic harm, and would require it to meet a demanding burden of proof in court. Worse, AEGIS would also prevent states from picking up the slack. This design makes perfect sense if you're a company trying to minimize the risk that the government can overly constrain you. It makes little sense if you want American democracy to govern frontier AI effectively. This isn't the first time. The initial version of California's SB 1047 would have let the Attorney General sue to stop developers like Anthropic from imposing unreasonable risks of catastrophic harm. It would also have created a regulatory agency to help set safety standards. Anthropic objected, and helped get both provisions removed before ultimately offering some support for the bill (which I also genuinely appreciate). I’m not saying that Anthropic should have supported 1047 in its initial form; that’s a complex issue and I realize there were questions about whether earlier versions of the bill were less politically feasible. Still, in Anthropic's own words, it helped weaken the bill because "[t]here is a substantial risk that the bill and state agencies will simply be wrong about what is actually effective in preventing catastrophic risk, leading to ineffective and/or burdensome compliance requirements." The same goes for New York's RAISE Act, whose initial version also would have let the Attorney General sue to stop unreasonable risks of catastrophic harm. Anthropic credits itself as a supporter of RAISE, but it did not support the bill while it was in the legislature, including when it nearly died. After the bill passed the legislature, Anthropic publicly critiqued it, saying "Any state proposals should be narrowly focused on transparency and not overly prescriptive." Why does Anthropic consistently take the risk of clumsy government action far more seriously than its own repeated warnings that companies like Anthropic will literally destroy the world by racing to RSI? 4. Even if AEGIS was an honest mix-up, that is itself a serious problem. Your response frames AEGIS as an early draft and a miscommunication. Perhaps you would have strengthened it and narrowed its preemption. (That wouldn't explain Anthropic's interventions to weaken SB 1047 or RAISE, but leave that aside.) Suppose that's exactly right. A federal framework for governing RSI is one of the highest-stakes documents your company will touch. Your post admits Anthropic "could have handled [these matters] better," but its tone ("This kind of thing happens a lot in large organizations") doesn't reflect the gravity of this issue. Anthropic talks about the need for “operational excellence” in critical areas, and I have a hard time believing that a similarly cavalier attitude would be tolerated while Anthropic was negotiating a key compute deal or preparing its next large pre-training run. In short The justification for racing rests on Anthropic fighting for strong governance, and Anthropic's record on regulation points the other way. Together with its breakneck racing, that fits a much darker picture of the company than the one that seems to prevail among your employees. I hope that Anthropic reflects on this picture, supports stronger regulation, exercises extraordinary caution when advocating for weak regulation, and doubles down on its founding purpose of helping the world safely navigate transformative AI together, instead of prioritizing Anthropic’s ability to win a race that imposes intolerable risks on everyone alive. open on X1 likes · 0 reposts
- Pretty sure they were just trying to convey “we won’t slow down unilaterally” but just in softer language (which is not a new position, see e.g. the related RSP changes a while back. Whether it is defensible is a separate Q)Wait so Anthropic's s-1 says that the calls to "pace the frontier" don't have anything to do with its own model release timelines?!? Am I reading that wrong? Or is this...just 'you can just say things and they mean whatever you want' open on X7 likes · 0 reposts
- Anthropic has filed an S-1 registration statement with the SEC related to its proposed IPO. Read more: anthropic.com/news/s1-sec Legal disclaimer: A registration statement relating to these securities has been filed with the SEC but has not yet become effective. These securities may not be sold, nor may offers to buy be accepted, prior to the time the registration statement becomes effective. This post does not constitute an offer to sell or the solicitation of an offer to buy these securities, nor shall there be any sale of these securities in any state or jurisdiction in which such offer, solicitation or sale would be unlawful prior to registration or qualification under the securities laws of any such state or jurisdiction.1,966 likes · 622 reposts
- Anthropic's S-1 is the first time I've been reading a document and thought 'actually on reflection some of these sections weren't worth it, they could have used more AI.'30 likes · 1 reposts
- I don't know what's the problem. Zitron was absolutely correct in saying "This company is not worth $60 billion."Some entertaining highlights from wheresyoured.at/wheres-the-mon… 20 months ago: - "Anthropic has projected (made up) that it will make at least $12 billion in revenue in 2027" - "This company is not worth $60 billion. Anthropic has raised $14.7 billion to create an also-ran Large Language Model company" - "Anthropic, much like OpenAI, has categorically failed to productize its Large Language Model" - "Its CEO is also a sleazy conman who, like Sam Altman, continually promises that his company's AI systems will become powerful and autonomous in a way that they have never shown any possibility of becoming." - "Any investor in Anthropic needs to seriously consider what it is they're investing in. Its products are both unpopular and commoditized, and it lost $5.6 billion last year! Stop dancing around this fact! Stop it!" - "there’s a lot of demand to put AI in stuff and some demand to buy stuff with AI on it by enterprises buying software, but little evidence to suggest significant user adoption or usage, I’d argue because Large Language Models do not lend themselves to features that provide meaningful business returns." - "When a company uses the term “agent,” they are intentionally trying to be deceitful, because the term “agent” means “autonomous AI that does stuff without you touching it.” The problem with this definition is that everybody has used it to refer to “a chatbot that can do some things while connected to a database,” which is otherwise known as a chatbot." - "It isn’t working. The users aren’t there. The revenue isn’t there. The best time to stop this was two years ago, and the next best time is as soon as humanly possible. I have said that generative AI is a group delusion in the past, and I repeat that claim today." - "we are approaching the limit of what the transformer-based architecture can do, if we haven’t already reached it. No amount of beating off about test-time compute and connecting Large Language Models to other Large Language Models is going to create a new use case for this technology" - "generative AI is OpenAI. The consumer adoption of this software has completely failed, and appears to be going nowhere fast. ChatGPT is sustained entirely on deranged, specious hype drummed up by a media industry" ... and my favorite ... - "I will keep writing this stuff until I’m proven wrong." open on X15 likes · 2 reposts
- I think it's only moderately embarrassing for Zitron that he got this stuff wildly wrong. It's valuable to have someone making the bear case! But it's extremely frustrating that he hasn't admitted this was completely incorrect and explained where he thinks he went wrong.Some entertaining highlights from wheresyoured.at/wheres-the-mon… 20 months ago: - "Anthropic has projected (made up) that it will make at least $12 billion in revenue in 2027" - "This company is not worth $60 billion. Anthropic has raised $14.7 billion to create an also-ran Large Language Model company" - "Anthropic, much like OpenAI, has categorically failed to productize its Large Language Model" - "Its CEO is also a sleazy conman who, like Sam Altman, continually promises that his company's AI systems will become powerful and autonomous in a way that they have never shown any possibility of becoming." - "Any investor in Anthropic needs to seriously consider what it is they're investing in. Its products are both unpopular and commoditized, and it lost $5.6 billion last year! Stop dancing around this fact! Stop it!" - "there’s a lot of demand to put AI in stuff and some demand to buy stuff with AI on it by enterprises buying software, but little evidence to suggest significant user adoption or usage, I’d argue because Large Language Models do not lend themselves to features that provide meaningful business returns." - "When a company uses the term “agent,” they are intentionally trying to be deceitful, because the term “agent” means “autonomous AI that does stuff without you touching it.” The problem with this definition is that everybody has used it to refer to “a chatbot that can do some things while connected to a database,” which is otherwise known as a chatbot." - "It isn’t working. The users aren’t there. The revenue isn’t there. The best time to stop this was two years ago, and the next best time is as soon as humanly possible. I have said that generative AI is a group delusion in the past, and I repeat that claim today." - "we are approaching the limit of what the transformer-based architecture can do, if we haven’t already reached it. No amount of beating off about test-time compute and connecting Large Language Models to other Large Language Models is going to create a new use case for this technology" - "generative AI is OpenAI. The consumer adoption of this software has completely failed, and appears to be going nowhere fast. ChatGPT is sustained entirely on deranged, specious hype drummed up by a media industry" ... and my favorite ... - "I will keep writing this stuff until I’m proven wrong." open on X12 likes · 0 reposts
- If we don't all die, we may at least see stocks go up!
12 likes · 0 reposts - I see this S-1 is going to be informative.
10 likes · 0 reposts - https://t.co/pJIjMV2Fbv8 likes · 0 reposts
- WSJ op-ed: "For investors mulling the Anthropic share offering, a 10% chance of extinction may be a 90% win for your portfolio"
6 likes · 2 reposts - search existential@_nathancalvin: 3 Mentions of catastrophic + existential risks in Ant S1: • Investors may be hurt because Anthropic models may behave in unintended/catastrophic ways • Anthropic may prioritize safety in a way that hurts investors • Anthropic may receive bad media coverage about risks open on X4 likes · 0 reposts
- And by pace the frontier, we mean...
3 likes · 1 reposts - If you find one Ant in your stock exchange... there is probably only going to be one Ant total because companies generally don't want to duplicate ticker symbolsAnthropic has officially filed for an IPO under the “ANT” symbol. open on X3 likes · 0 reposts
- Everybody Claude Now.
2 likes · 1 reposts - WSJ says Broadcom CEO Hock Tan directly intervened to get Anthropic's debt deal pushed though.
SCOOP: Broadcom CEO Hock Tan was getting impatient. He needed lenders to commit $40 billon in the largest AI debt deal ever—so Anthropic could afford TPUs—but some were getting cold feet. In the end, Broadcom’s CEO held 11th-hour talks with bank executives over the weekend to get them to finalize their commitments. Broadcom also agreed to higher interest rates on the loans, sweetening the deal. With @ajsaeedy @ mattwirz for @WSJ open on X2 likes · 0 reposts - Oh no!
2 likes · 0 reposts - Drink.
2 likes · 0 reposts - There’s a common argument that Anthropic and OpenAI already have every incentive to crack down on distillation, since it hurts their competitiveness If a Chinese lab can just distill your frontier model, it undercuts you on price and eats into your margins But that’s not the full story... Anthropic told investors that 5-7% of its revenue comes from customers who are DISTILLING ITS MODELS So in the near term, allowing distillation pumps up their API revenue. But in the long term, it erodes their lead in the AI raceDistillation attacks accounted for c.$1.65-2.3bn of Anthropic's first nine months' revenue in 2026. The S1 confirms Anthropic's 1H revenue was $16.2bn, and FT reporting from earlier in October suggests Q3 revenue was $16.8bn. The S1 states that c.5% to 7% of Anthropic's revenue in the first 9 months of this year was derived from customers who were subsequently found to have violated Anthropic's "usage policy or commercial terms." If essentially all of this revenue came from illicit distillation, applying Fable 5’s list prices of $10:50/m tokens and assuming a 1:17 input ratio implies c.33–46 trillion output tokens, after paying for inputs. The input:output ratio is informed by tokenizing the two math and coding samples in the DeepSeek-R1 paper, which are from a dataset IFP says is "likely similar to what Chinese companies can extract when distilling US models behind an API." This is likely a lower-bound estimate because cheaper models or discounted access would yield more output tokens for the same spend, and we know Anthropic models cheaper than Fable 5 were distilled from, as well as Fable 5 itself. ifp.org/how-to-increas… sec.gov/Archives/edgar… ft.com/content/11632d… cisa.gov/news-events/cy… arxiv.org/html/2501.1294… open on X1 likes · 0 reposts
- Not just Anthropic…1 likes · 0 reposts
- Oh interesting. My initial thought was that this clause was added to calm investors who were worried that Anthropic pushing for Pacing the Frontier would slow down development and thus reduce share price On reflection, ig it is kinda confusing what exactly they mean here1 likes · 0 reposts
- Source: from their S-1 filing for IPO sec.gov/Archives/edgar…1 likes · 0 reposts
- Wonder what Argon’s chain of thought would say hereGoogle Gemini is very against Anthropic IPO lol open on X0 likes · 0 reposts
- Seems correct0 likes · 0 reposts
- source: Anthropic's S-1 filing sec.gov/Archives/edgar…0 likes · 0 reposts
- false imprisonment perhaps0 likes · 0 reposts
@_nathancalvin policy · 
@daverbanerjee policy · 
@katjagrace research · 
@miles_brundage policy · 
@anthropicai org · 
@thezvi commentary · 
@boazbaraktcs lab · 
@kelseytuoc press · 
@peterwildeford policy · 
@andrewcurran_ commentary · 
@aaronscher research ·