AVERI pilot report finds Amazon's internal safety evals consistent with its public claims
AVERI released a pilot report reviewing 800 prompts sampled from Amazon's internal safety evaluation suite, finding Amazon's public statements about its Nova 2 testing consistent with the sample and prompting further improvements. Miles Brundage, who collaborated on the project, shared the results, framing it as rare independent assessment of companies' self-developed safety tests. Amazon Science published a companion post on the partnership.
All posts · 1, most important first
- Excited to share the results of a recent pilot project with Amazon! Companies use private safety evaluations to make decisions with big public consequences. Our review led to several improvements in Amazon’s internal evals. Thanks to @rahul1987iit for collaboration on this:AI companies make safety calls based in large part on internal tests they develop themselves. It’s rare for there to be independent assessment of the quality of those tests. Earlier this year, @Amazon and @AVERIorg agreed to a pilot project in which AVERI assessed the quality of Amazon’s internal safety evaluations. Today we’re releasing the result of that independent review. As a result of this project, Amazon is changing how it tests future models. Our review of Amazon’s internal AI safety evaluations confirmed several strengths such as topical diversity, and we found the evaluations to be consistent with Amazon’s public statements on them. At the same time, we found areas for improvement: On attack sophistication, some prompts represented relatively straightforward attack vectors that would test safety filters rather than deeper model reasoning. On evaluation signal quality, we identified a concern that superficial cues, rather than safety reasoning, may drive some model refusals, so passing these evaluations might not indicate robust safety behavior in novel situations. On linguistic and stylistic diversity, some multilingual prompts were included, but prompts across risk categories were predominantly in English, and some prompt structures were reused across scenarios. On evaluation freshness, a small number of prompts contained temporal references that could become outdated. In response, Amazon reports four changes for future internal evaluations: Refreshing its test sets to remove redundant prompts and add more challenging adversarial scenarios; Introducing a tiered difficulty calibration framework; Expanding multilingual coverage across all risk categories; and Establishing a quarterly review schedule to identify staleness and incorporate emerging attack patterns. Why does this matter? A passing grade on an easy test doesn’t tell us much on its own. As the stakes of AI development and deployment grow, we think that this kind of independent evaluation of internal evaluations (“metaevaluation”) will grow in importance. We appreciate Amazon opening their internal evaluations up to outside review. We hope more companies take this step to help make their models safer. While no clear standards exist for metaevaluation yet, @AVERIorg is working to change that. We look forward to sharing more pilot results in the coming months and will continue our work developing standards with institutions like @aievalforum. Read AVERI’s full blog post here: averi.org/ourwork/averi-… Read Amazon’s blog post on the collaboration here: amazon.science/blog/strengthe… open on X13 likes · 1 reposts
@miles_brundage policy ·