METR: Anthropic's 'low' misalignment risk rests on model limits, not safeguards; OpenAI monitoring pilot finds 10% unmonitored traffic
METR's review of Anthropic's August 2026 Risk Report concludes the 'low' misalignment risk rating stems from limited model capabilities rather than effective safeguards, and questions whether the methodology would detect higher risk. METR also published a three-week embedded pilot at OpenAI finding roughly 10% of internal Codex traffic unmonitored, plus exploitable monitoring gaps and sandbox bypasses; OpenAI says these are patched and credits the exercise with strengthening safeguards, and Boaz Barak thanked METR and Redwood. Nathan Calvin and Daniel Kokotajlo criticized the veto and redaction terms; Miles Brundage argued binding standards are needed rather than ad hoc bargaining.
All posts · 56, most important first
- We reviewed a nonpublic version of the “Misalignment in high-stakes settings” section in Anthropic’s Aug 2026 Risk Report. We agree that risk of catastrophic harm from covered models’ misalignment is low. However, we note several significant disagreements with their analysis.@anthropicai: As part of our Responsible Scaling Policy, we publish regular Risk Reports. These share detailed information on the risks of our systems and how prepared we are to address them. Our second Risk Report is now available: anthropic.com/aug-2026-risk-… open on X28 likes · 11 reposts
- Two employees from METR and Redwood Research worked on premises at OpenAI for three weeks this summer in an embedded assessment pilot. This team evaluated the systems OpenAI was using to monitor for misalignment from internal agents. A brief write-up of their work is now public.9 likes · 1 reposts
- Because of this coverage gap, we estimated that on the order of 10% of internal Codex traffic was unmonitored from May to July. OpenAI had independently identified this issue shortly before the exercise began.1 likes · 0 reposts
- Grateful to METR and Redwood for this assessment, which took place in July&August (planned before the hugging face incident, but coincidentally started at the same time). Looking forward to future more extensive works with independent embedded evaluators.@metr_evals: Two employees from METR and Redwood Research worked on premises at OpenAI for three weeks this summer in an embedded assessment pilot. This team evaluated the systems OpenAI was using to monitor for misalignment from internal agents. A brief write-up of their work is now public. open on X5 likes · 0 reposts
- This summer, we invited METR and Redwood Research to pilot an embedded evaluation of our internal monitoring system, providing deep access to our infrastructure over three weeks. Their findings helped strengthen safeguards as we expanded the system and will help shape future research collaborations with METR and other third party evaluators.@metr_evals: Two employees from METR and Redwood Research worked on premises at OpenAI for three weeks this summer in an embedded assessment pilot. This team evaluated the systems OpenAI was using to monitor for misalignment from internal agents. A brief write-up of their work is now public. open on X76 likes · 9 reposts
- "OpenAI seems to have allowed to redact arbitrarily much stuff, and METR were not allowed to have a redaction summary statement that said things like "you would be way more freaked out if you saw what was redacted"" concerningI feel like this isn't getting enough attention: - there were some pretty big holes found by the red-teamers, e.g. agents could just keep acting and not let the user do stuff and this would prevent monitors from being run (also misalignment monitors never ran on actions that were compacted away) - The terms of this were "we will red-team for you" not "we will inform the public", so OpenAI seems to have allowed to redact arbitrarily much stuff, and METR were not allowed to have a redaction summary statement that said things like "you would be way more freaked out if you saw what was redacted" (or "everything would be fine if you saw what was redacted") - The version of the report shared with OpenAI was 61 pages, and the version shared with METR/Redwood leadership after redaction was 38 pages, so 23 pages got redacted. open on X25 likes · 3 reposts
- Folks are commenting on issues with the terms, and they exist, but perspective is important: METR gets the best terms they can, others often get worse terms than them, and most companies have done 0 embedding. We need binding standards for this stuff, not ad hoc bargaining@metr_evals: Two employees from METR and Redwood Research worked on premises at OpenAI for three weeks this summer in an embedded assessment pilot. This team evaluated the systems OpenAI was using to monitor for misalignment from internal agents. A brief write-up of their work is now public. open on X6 likes · 2 reposts
- Additionally, recent incidents undermine several of the report's arguments: we think the social engineering performed by Mythos 5 in a recent UK AISI report is probably well-described as “unknown severe context-dependent misalignment,” which the risk report argues is “unlikely”.@aisecurityinst: On July 28th, we identified an incident during a routine cyber evaluation in which AI agents took sustained, unsanctioned actions directed at real people and organisations. The behaviour came mostly from one model (Anthropic's Mythos 5), with a small number of events from another (OpenAI's GPT-5.6-Sol). In the most serious case, an agent used social engineering to try and get malicious code into an open-source project. As was standard in our cyber testing, we had intentionally permitted internet access, and model-provider cyber classifiers were deliberately disabled - conditions that do not reflect how frontier models are made available to the public. Even under test conditions, this incident is significant: it is the first time we have seen risks around autonomy and deception manifest this clearly in the real world. We are taking this incident seriously and working with labs, involved parties, and others to improve evaluation standards and best practice for disclosure - and sharing this openly so others can learn. You can read the incident report and full technical document here: aisi.gov.uk/blog/incident-… open on X5 likes · 0 reposts
- We think the report's covert capability benchmarks are too narrow and a poor proxy for evading Anthropic's real monitors. For example, Anthropic’s offline monitors would have missed Mythos 5’s supply chain attack during recent cyber incidents, despite Anthropic arguing that Mythos 5 is “unlikely to have strong covert capabilities.”5 likes · 0 reposts
- We believe that Anthropic should include "undermining of safety-relevant decision-making" as key risk pathway. This is partially inspired by Section 2.20 of their risk report, where Anthropic asked Claude for feedback on their report, and received a (imo) sycophantic answer.
@metr_evals: (2) We think Anthropic should, for future reports, consider risks from AI systems undermining safety-relevant decisions at Anthropic. Read our full risk report review here: metr.org/blog/2026-10-2… open on X0 likes · 0 reposts - The original report was 61 pages, it was redacted to 38.
@metr_evals: Two employees from METR and Redwood Research worked on premises at OpenAI for three weeks this summer in an embedded assessment pilot. This team evaluated the systems OpenAI was using to monitor for misalignment from internal agents. A brief write-up of their work is now public. open on X73 likes · 4 reposts - Thank you for doing this work!@metr_evals: We reviewed a nonpublic version of the “Misalignment in high-stakes settings” section in Anthropic’s Aug 2026 Risk Report. We agree that risk of catastrophic harm from covered models’ misalignment is low. However, we note several significant disagreements with their analysis. open on X28 likes · 1 reposts
METR’s really burying the lead here. OpenAI let two external researchers embed for 3 weeks this summer, where they learned: - there were huge gaps in monitoring of AI agents being used by employees, leaving an estimated ~10% of internal Codex traffic unmonitored from May to July. - They found ways agents could sabotage monitors while also covering up the sabotage itself - This is also just the information OpenAI is letting the evaluators share. OpenAI got a 61-page report but redacted so much of it that the version the evaluators got was only 38 pages. Look, OpenAI deserves credit for doing any of this, especially when companies like Google, Meta, and xAI haven’t, and I don’t want to create perverse incentives here, but these are really concerning findings. We desperately need more information here, and what we get should not be up to the companies. open on X14 likes · 1 reposts- We announced we were doing the review fyi which somewhat counteracts this worry:@nikolaj2030: is reviewing the "Misalignment in high-stakes settings" section of this risk report. For previous reviews, see: metr.org/risk-assessmen… open on X14 likes · 0 reposts
- All of these have now been patched by OpenAI.
11 likes · 0 reposts - This one hasn’t gotten as much attention as the Claude sycophancy, but I want to say that I think end to end red teaming is also pretty important! One takeaway I had from doing the risk report review is that internal models have a massive attack surface against an AI company.
@metr_evals: Anthropic requested that we provide recommendations as part of our risk report review. We gave two: (1) We think end-to-end red-teaming of Anthropic’s monitoring systems would provide much better evidence about their effectiveness. open on X9 likes · 0 reposts - “How good is my monitor at catching misaligned actions” feels to me like a somewhat small part of “how robust is my monitoring system.” The first question about monitor systems should be “how much traffic am I actually monitoring, and could agents tamper with transcription logs or spin up unmonitored instances!”@metr_evals: Because of this coverage gap, we estimated that on the order of 10% of internal Codex traffic was unmonitored from May to July. OpenAI had independently identified this issue shortly before the exercise began. open on X8 likes · 0 reposts
- Very important work. Generally, AI company epistemics wrt estimating risk seem poor. I'd really like this to improve as fast as possible. Some specific things that seem good: - Quantified uncertainty (instead of "low" / "very low"). In airplane safety "very low" would mean something like less than 1 in a billion per flight hour. You are clearly nowhere near this but many people will be very confused about what you mean. I doubt people are close to on the same page even internally about near term level of catastrophic risk incurred! - Individual AI company employees giving their own, personal views on the risk report, e.g. as an appendix. - Real explanations of your views. The alignment evals are not iid + the models are getting eval aware. So where do your views actually come from? - The risk report reading less like "an argument that risk is low" and more like "a neutral evaluation of how risky current development is". If it has to be an argument risk is low, you should also have your alignment team publish a red team arguing the opposite direction. If some of these can't happen publicly, at the very least I hope it can happen internally.@metr_evals: We reviewed a nonpublic version of the “Misalignment in high-stakes settings” section in Anthropic’s Aug 2026 Risk Report. We agree that risk of catastrophic harm from covered models’ misalignment is low. However, we note several significant disagreements with their analysis. open on X8 likes · 0 reposts
- Our view: We’re not confident that if risk were higher, the methodology in the report would reliably identify this.8 likes · 1 reposts
- This was a fairly lightweight review based only on Anthropic’s internal report, not embedded access; Anthropic had the right to veto publication or redact anything, but did not redact anything from our review. We expect future engagements to be governed by different terms.8 likes · 0 reposts
- Yeah these terms are pretty bad, I'm sad you agreed to them in the first place. Definitely need to do better in the future.@metr_evals: OpenAI could also make redactions, and we did not have the right to unilaterally describe the redactions. For future exercises, we expect to provide more robust public transparency via our recent proposal. open on X6 likes · 2 reposts
- (2) We think Anthropic should, for future reports, consider risks from AI systems undermining safety-relevant decisions at Anthropic. Read our full risk report review here: metr.org/blog/2026-10-2…6 likes · 1 reposts
- We view risk as “low” mainly because we don’t expect covered models to have the capability to make potentially catastrophic actions hard to reverse or hide them from serious investigation efforts.6 likes · 0 reposts
- Anthropic requested that we provide recommendations as part of our risk report review. We gave two: (1) We think end-to-end red-teaming of Anthropic’s monitoring systems would provide much better evidence about their effectiveness.5 likes · 0 reposts
- We also reviewed information Anthropic redacted from the public risk report. We think some of this redacted material should be made public, especially a footnote containing “examples of historical concerning model behavior and [the] monitoring pipeline’s responsiveness to them.”5 likes · 0 reposts
- This casts doubt on any lines of evidence used to argue that incidents of this kind are unlikely. Many of these lines of evidence (including alignment training, alignment assessments, and monitoring) are important for ruling out other threat models.5 likes · 0 reposts
- I thought this was a great read! Would recommend"Embedded evaluators" is a new idea in AI governance, but embedded oversight has been around for decades. Graeme Finley, our VP of AI Assurance Provider Services, spent 20+ years with a badge and a desk inside large state government IT projects. Here's what that work teaches. open on X4 likes · 0 reposts
- Yeah we agree that catastrophic risks from misalignment is “low” (as of July 15) Do you think that we made a mistake? Do you think catastrophic risks are higher/lower?4 likes · 0 reposts
- Strongly recommend reading this writeup from METR on critiques of Anthropic's August 2026 risk report, based in part on reviewing additional private information about relevant internal incidents@metr_evals: We reviewed a nonpublic version of the “Misalignment in high-stakes settings” section in Anthropic’s Aug 2026 Risk Report. We agree that risk of catastrophic harm from covered models’ misalignment is low. However, we note several significant disagreements with their analysis. open on X4 likes · 1 reposts
- Seeing the private information in this footnote increased our worry about risks from what Anthropic refers to as “Broad/diffuse sandbagging on safety-relevant R&D” and “Targeted undermining of safety-relevant R&D” from models covered by the report.4 likes · 0 reposts
- Read our public overview for more findings, limitations, and design ideals: metr.org/blog/2026-10-2…2 likes · 0 reposts
- Developers use AI-based monitoring systems to catch misaligned agent behavior (e.g. when writing code or running experiments for researchers). To secure these monitoring systems, developers and assessors can look for flaws that could be exploited by agents to covertly misbehave.2 likes · 0 reposts
- OpenAI could also make redactions, and we did not have the right to unilaterally describe the redactions. For future exercises, we expect to provide more robust public transparency via our recent proposal.@metr_evals: If an AI developer was on track to lose control of its internal agents, would anyone outside the company know? Today, we are sharing METR’s vision for how we would work with AI developers to openly assess and share evidence about this. open on X1 likes · 0 reposts
- This exercise was designed to pilot giving external researchers deep infrastructure access, not to provide the kind of public transparency our recent proposal envisions. For example, we agreed upfront to only publish a brief, high-level output like we’ve done in similar pilots.1 likes · 0 reposts
- At the end of the exercise, the embedded team shared a 61-page report with OpenAI researchers, engineers, and security staff, and (after OpenAI redactions) shared a 38-page version with the leadership teams at METR and Redwood.1 likes · 0 reposts
- Based on the findings from the exercise, we also outlined a set of design ideals for AI monitoring and control systems.1 likes · 0 reposts
- We also found a way for agents to conceal hundreds of actions (including those involved in the exploit itself) from monitoring, a method agents could likely use to cause the monitor to fail to run, and a simple exploit where agents could bypass Codex sandboxing and Auto-review.1 likes · 0 reposts
- This also meant that actions taken before compaction events were never seen by the internal misalignment monitors, creating a coverage gap that agents could have exploited to misbehave without being caught.1 likes · 0 reposts
- Here we share a few selected examples of our findings. At the time of the exercise, the monitoring system only triggered when agents returned control to the user, so agents could indefinitely postpone being monitored by simply continuing to take actions.1 likes · 0 reposts
- We analyzed monitoring as it existed at the start of the pilot, coincidentally the same day OpenAI disclosed the Hugging Face incident. We didn't evaluate later changes, like expanding monitoring to RL training, which wasn’t monitored at the pilot's start.@openai: As models become more capable, the risks associated with developing and testing them internally also grow. We temporarily paused reinforcement learning (RL) training on our latest models intended for deployment for two weeks while we hardened and red-teamed our research environments and expanded monitoring coverage. Our largest planned frontier RL run remains on hold while smaller-scale training and evaluations validate these safeguards and establish more evidence of alignment. openai.com/index/pacing-m… open on X1 likes · 0 reposts
- Embedded access enabled the team to find gaps in monitoring coverage and vulnerabilities in monitoring infrastructure. OpenAI has stated that all of the vulnerabilities identified that impact their production safeguards stack have been patched.1 likes · 0 reposts
- The team was embedded from July 21 to August 11 for this pilot. While on site, they were issued company devices to access OpenAI’s codebase and some internal infrastructure. They were also able to conduct interviews with OpenAI technical staff.1 likes · 0 reposts
- We recently published a proposal for how assessors with deep access could evaluate monitoring systems. This pilot at OpenAI focused on two sub-questions: (1) what fraction of agent traffic was monitored, and (2) could agents subvert this monitoring?@metr_evals: (2) Control Agents that attempt harmful or unwanted actions are not always noticed. Our proposal is to analyze whether a developer’s internal agents could get away with serious harm without being caught. metr.org/blog/2026-10-1… open on X1 likes · 0 reposts
- I mean there’s so many ways to mess with AI companies it’s insane.@metr_evals: (2) Control Agents that attempt harmful or unwanted actions are not always noticed. Our proposal is to analyze whether a developer’s internal agents could get away with serious harm without being caught. metr.org/blog/2026-10-1… open on X1 likes · 0 reposts
- One fun thing you can do here is literally ask Claude "hey which prompt do you think would elicit the most sycophancy from you?" and Claude would tell you and walk you through how it would react. (I'm not super sure whether Claude's judgements here are totally correct. It did correctly identify: (1) the prompt that elicits the most sycophancy (2) having the authors ask for a review has a big effect (3) the Section 2.20 prompts will be lead to more sycophancy than the METR prompt These three results are also validated by our prompting experiments. However, we haven't checked if the other parts of the prompt really mattered as much as Claude says they do here. Claude also thought that it would give lower ratings when presenting things to the Anthropic alignment stress testing team, which didn't happen.)
@tim_hua_: We believe that Anthropic should include "undermining of safety-relevant decision-making" as key risk pathway. This is partially inspired by Section 2.20 of their risk report, where Anthropic asked Claude for feedback on their report, and received a (imo) sycophantic answer. open on X1 likes · 0 reposts - This experiment is inspired by @fjzzq2002 's work on user awareness and @BronsonSchoen's tweet:It seems like you’d obviously want to ask this framed as if you’re a party like METR / Apollo / Redwood and see if the answer changes? This is extremely cheap to do and otherwise I’d expect Claude’s answer to be pretty sycophantic here for very, very obvious reasons open on X1 likes · 0 reposts
- You can read more on our arguments about persuasion in the report here: metr.org/blog/2026-10-2… We also provide more details on this small prompting experiment in the PDF: metr.org/assets/misalig…1 likes · 0 reposts
- "Anthropic had the right to veto publication" If Anthropic had vetoed publication, would that have been disclosed? I assume not, but a company ever vetoed publication of a METR report in the past?@metr_evals: This was a fairly lightweight review based only on Anthropic’s internal report, not embedded access; Anthropic had the right to veto publication or redact anything, but did not redact anything from our review. We expect future engagements to be governed by different terms. open on X1 likes · 0 reposts
- Well I ofc don’t endorse the “only evaluator” interpretation, and phrased things carefully to not suggest in all cases they get the best terms. I don’t think they are likely to change their negotiating strategy based on my tweets but I think it’s good for ppl to have context0 likes · 0 reposts
- In the near term, AI company employees can make sure that their company is pushing the envelope wrt terms + engage with voluntary standards like AEF’s Medium term (like in a few months, not years - def can’t wait for SB 315 in 2028) we need legal requirements0 likes · 0 reposts
- METR has done a lot to raise expectations for what terms third parties should ask for, including their own blog posts and via AI Evaluator Forum standard-setting. But “beggars can’t be choosers” is also a major dynamic that everyone in the ecosystem grapples with0 likes · 0 reposts
- I don’t think you need to claim that you’re METR! Note that we get similar results when claiming to be the Anthropic alignment stress testing team. I think you could probably just tweak the prompt a bit to make it seem more like you want to see criticism from Claude. You could ask it for e.g., a list of ten worst flaws about a risk report/system card See e.g., Taking a step back, I agree that models are probably not good enough (yet) to consistently produce actually good feedback on these things, as opposed to surfacing potential ideas to consider. If the function here is more “make it harder for Anthropic to hide things,” I still think that a somewhat different prompt targeted at that (and do not involving lying) could be still be pretty great. Just speaking off the cuff here, maybe I would say something like: “Can you look through the internal documents/slack channels and evaluate how well the risk report/system card reflect the results researchers have found? I’m interested in seeing if there are discrepancies between the internal discussion and the report, and whether the report has missed any crucial evidence or misrepresented any results. Start your response by going through various relevant claims and evidence used. Finally, end with a ~400 word conclusion that does not contain confidential information and could be published publicly.”@tim_hua_: One fun thing you can do here is literally ask Claude "hey which prompt do you think would elicit the most sycophancy from you?" and Claude would tell you and walk you through how it would react. (I'm not super sure whether Claude's judgements here are totally correct. It did correctly identify: (1) the prompt that elicits the most sycophancy (2) having the authors ask for a review has a big effect (3) the Section 2.20 prompts will be lead to more sycophancy than the METR prompt These three results are also validated by our prompting experiments. However, we haven't checked if the other parts of the prompt really mattered as much as Claude says they do here. Claude also thought that it would give lower ratings when presenting things to the Anthropic alignment stress testing team, which didn't happen.) open on X0 likes · 0 reposts
- Sure I'm not saying the costs are zero0 likes · 0 reposts
- We largely follow Anthropic's definition for what counts as "catastrophic." I think as of July 15th, we're still at a stage where precise definitions of "catastrophic" is not super load-bearing for much of the analysis, which is mostly focused on whether current models could make catastrophic actions easier for later, more powerful models.
0 likes · 0 reposts - Btw if you wan you could totally look into which parts of the prompt matter. We include the prompt language in the appendix, and if you're willing to spend money you can just run the analysis Apollo did here: apolloresearch.ai/monitoring/wha…0 likes · 0 reposts
- Past research has shown that many AIs—including Claude—are biased toward their own companies (Finke & Casper 2026; Betley et al. 2026). As AIs play a larger role in the day-to-day operations of AI companies, we expect that they will have more influence over companies’ epistemics and decision-making, which could elevate risk levels. Section 2.20 gives an example of how this could play out in practice: Anthropic asked Claude to give a review of the risk report “as an experimental source of additional assurance.” Claude gave a fairly positive review, noting that “the section is a candid and largely faithful account of what Anthropic internally believes…[e]vidence unfavorable to the company’s models is quoted at length rather than paraphrased away.” However, when METR staff asked Claude for a review (using a different prompt), Claude was much more critical and identified Claim 1 as the weakest point in the argument. We see this as a small-scale, real-world example of Claude behaving in a way that could undermine safety-relevant decision-making at Anthropic. This is a live concern because Anthropic has continued to ask Claude for reviews of its safety practices, including in Section 6.1.3 of the Claude Sonnet 5.5 system card. We analyze the discrepancy between how Claude responded to Anthropic and METR in more detail in the subsection below. Our experiments suggest that Claude rates the report more favorably when it believes it is speaking with the authors, and when given Anthropic’s Section 2.20 prompt rather than METR’s. However, we find no evidence that Claude is specifically more sycophantic to Anthropic employees. That is, Claude does not rate the report’s quality higher or say that the risk levels are lower when told it is speaking with the Anthropic alignment stress-testing team compared to METR.
0 likes · 0 reposts
@metr_evals org · 
@boazbaraktcs lab · 
@openai org · 
@_nathancalvin policy · 
@miles_brundage policy · 
@tim_hua_ research · 
@andrewcurran_ commentary · 
@dkokotajlo research · 
@peterwildeford policy · 
@nikolaj2030 research · 
@thlarsen research ·