US Agencies Release Guidance for Managing AI Agent Control Incidents
US government agencies (NSA, NIST, CISA) define and mandate controls for "SI control incidents" involving autonomous AI agents. Companies are required to implement technical controls, including rapid kill mechanisms and external audits of monitoring systems. Structured risk assessments must be completed and approved by senior leadership before any deployment or training runs.
All posts · 2, most important first
- New US government guidance from CAISSI + NSA + CISA: The US government defines an "SI control incident" where an agent takes significant uninstructed actions that break operator policies, controls, or oversight. Near-misses are in scope too. Companies are then asked to: - have technical controls, including rapid kill and containment mechanisms and red-teaming. - Bring in independent external auditors to verify that control and monitoring systems work. - Write a structured risk assessment before training runs, evals, or deployments, and proceed only if senior leadership is satisfied with the risk, informed by external evaluators where appropriate. - Monitor every layer with both programmatic and SI-based monitors, covering trajectories, reasoning, and outputs. - Make sure agents can't touch the monitors or logs - Use monitor models from different model families. - In the event of an incident, preserve weights, intermediate checkpoints, environments, and rejected tool calls. - Keep tamper-resistant logs with reliable timestamps, including reasoning traces. - If you use models to help investigate, cross-check with humans and with models outside the incident model's family. - Shutdown policies calibrated to SI model capability and incident severity. - Depending on severity, pause training, evals, or deployment of similar agents until controls are re-verified. - When restarting, validate every containment layer and replay the incident trajectory to prove the new controls would catch it. - Document the decision to resume and the residual risk accepted. - Write an after-action report. - Notify anyone compromised, including intermediate victims, and share timelines and transcripts. - Report cyber incidents and near-misses to CISA. Share SI-specific incident and near-miss information with CAISSI and NSA SISC. - Give third parties enough access and documentation for independent review.NSA joined @NIST and @CISACyber in releasing guidelines for managing SI control incidents. Organizations utilizing SI are encouraged to adopt these practices. Read the joint report: media.defense.gov/2026/Oct/16/20… open on X3 likes · 1 reposts
- This looks pretty meaty, I like a lot of this. "Encouraged to". I find myself increasingly confused about what terms such as that means in a US AI policy context, can anyone explain? In the UK, I'd understand that as 'voluntary' but in the US, um, maybe "voluntary, but maybe we'll find some way to put you in a world of governance hurt if you don't do it?"@peterwildeford: New US government guidance from CAISSI + NSA + CISA: The US government defines an "SI control incident" where an agent takes significant uninstructed actions that break operator policies, controls, or oversight. Near-misses are in scope too. Companies are then asked to: - have technical controls, including rapid kill and containment mechanisms and red-teaming. - Bring in independent external auditors to verify that control and monitoring systems work. - Write a structured risk assessment before training runs, evals, or deployments, and proceed only if senior leadership is satisfied with the risk, informed by external evaluators where appropriate. - Monitor every layer with both programmatic and SI-based monitors, covering trajectories, reasoning, and outputs. - Make sure agents can't touch the monitors or logs - Use monitor models from different model families. - In the event of an incident, preserve weights, intermediate checkpoints, environments, and rejected tool calls. - Keep tamper-resistant logs with reliable timestamps, including reasoning traces. - If you use models to help investigate, cross-check with humans and with models outside the incident model's family. - Shutdown policies calibrated to SI model capability and incident severity. - Depending on severity, pause training, evals, or deployment of similar agents until controls are re-verified. - When restarting, validate every containment layer and replay the incident trajectory to prove the new controls would catch it. - Document the decision to resume and the residual risk accepted. - Write an after-action report. - Notify anyone compromised, including intermediate victims, and share timelines and transcripts. - Report cyber incidents and near-misses to CISA. Share SI-specific incident and near-miss information with CAISSI and NSA SISC. - Give third parties enough access and documentation for independent review. open on X0 likes · 1 reposts
@peterwildeford policy · 
@s_oheigeartaigh policy ·