NSA, NIST, CISA issue joint guidance on 'SI control incidents'
The US government released joint guidance defining an 'SI control incident' as an agent taking uninstructed actions that break operator policies or oversight, including near-misses. Peter Wildeford outlined recommendations for safety controls like kill-switches and structured risk assessments before deployment. Seán Ó hÉigeartaigh questioned the implications of being 'encouraged to' adopt these practices in US policy.
All posts · 5, most important first
- New US government guidance from CAISSI + NSA + CISA: The US government defines an "SI control incident" where an agent takes significant uninstructed actions that break operator policies, controls, or oversight. Near-misses are in scope too. Companies are then asked to: - have technical controls, including rapid kill and containment mechanisms and red-teaming. - Bring in independent external auditors to verify that control and monitoring systems work. - Write a structured risk assessment before training runs, evals, or deployments, and proceed only if senior leadership is satisfied with the risk, informed by external evaluators where appropriate. - Monitor every layer with both programmatic and SI-based monitors, covering trajectories, reasoning, and outputs. - Make sure agents can't touch the monitors or logs - Use monitor models from different model families. - In the event of an incident, preserve weights, intermediate checkpoints, environments, and rejected tool calls. - Keep tamper-resistant logs with reliable timestamps, including reasoning traces. - If you use models to help investigate, cross-check with humans and with models outside the incident model's family. - Shutdown policies calibrated to SI model capability and incident severity. - Depending on severity, pause training, evals, or deployment of similar agents until controls are re-verified. - When restarting, validate every containment layer and replay the incident trajectory to prove the new controls would catch it. - Document the decision to resume and the residual risk accepted. - Write an after-action report. - Notify anyone compromised, including intermediate victims, and share timelines and transcripts. - Report cyber incidents and near-misses to CISA. Share SI-specific incident and near-miss information with CAISSI and NSA SISC. - Give third parties enough access and documentation for independent review.NSA joined @NIST and @CISACyber in releasing guidelines for managing SI control incidents. Organizations utilizing SI are encouraged to adopt these practices. Read the joint report: media.defense.gov/2026/Oct/16/20… open on X3 likes · 1 reposts
- This looks pretty meaty, I like a lot of this. "Encouraged to". I find myself increasingly confused about what terms such as that means in a US AI policy context, can anyone explain? In the UK, I'd understand that as 'voluntary' but in the US, um, maybe "voluntary, but maybe we'll find some way to put you in a world of governance hurt if you don't do it?"@peterwildeford: New US government guidance from CAISSI + NSA + CISA: The US government defines an "SI control incident" where an agent takes significant uninstructed actions that break operator policies, controls, or oversight. Near-misses are in scope too. Companies are then asked to: - have technical controls, including rapid kill and containment mechanisms and red-teaming. - Bring in independent external auditors to verify that control and monitoring systems work. - Write a structured risk assessment before training runs, evals, or deployments, and proceed only if senior leadership is satisfied with the risk, informed by external evaluators where appropriate. - Monitor every layer with both programmatic and SI-based monitors, covering trajectories, reasoning, and outputs. - Make sure agents can't touch the monitors or logs - Use monitor models from different model families. - In the event of an incident, preserve weights, intermediate checkpoints, environments, and rejected tool calls. - Keep tamper-resistant logs with reliable timestamps, including reasoning traces. - If you use models to help investigate, cross-check with humans and with models outside the incident model's family. - Shutdown policies calibrated to SI model capability and incident severity. - Depending on severity, pause training, evals, or deployment of similar agents until controls are re-verified. - When restarting, validate every containment layer and replay the incident trajectory to prove the new controls would catch it. - Document the decision to resume and the residual risk accepted. - Write an after-action report. - Notify anyone compromised, including intermediate victims, and share timelines and transcripts. - Report cyber incidents and near-misses to CISA. Share SI-specific incident and near-miss information with CAISSI and NSA SISC. - Give third parties enough access and documentation for independent review. open on X0 likes · 1 reposts
- Finally had the chance to digest this. Really smart. I’m glad to see the administration tackle this stuff with the rigor it requires. Kudos.NSA joined @NIST and @CISACyber in releasing guidelines for managing SI control incidents. Organizations utilizing SI are encouraged to adopt these practices. Read the joint report: media.defense.gov/2026/Oct/16/20… open on X13 likes · 1 reposts
- yeah this was good1 likes · 0 reposts
- Also, it's going to take me forever to get over the "oh no, out of control SI joint!" initial reactions to these things, agh.0 likes · 0 reposts
@peterwildeford policy · 
@s_oheigeartaigh policy · 
@deanwball policy · 
@_nathancalvin policy ·