FAR AI releases FIBS and activation probes to detect AI agent sabotage
Researchers at FAR AI introduce FIBS and activation probes that detect AI agent sabotage and lies missed by traditional text monitors. The probes showed high AUROC on UK AISI hidden-goal models and reached 98.4% AUROC on Qwen3.5-397B.
All posts · 14, most important first
- New paper from our team, Telling FIBS by @OskarJohnH, @afspies, Tigist Diriba, @ARGleave and @ChrisCundy: probes that read a model's activations catch AI agents sabotaging their tasks. They match or beat monitors reading the agent's actions, without needing its reasoning, at a fraction of the cost. And they catch lies a text monitor can't. 1/8
1 likes · 0 reposts - Use FIBS and our probes: replicate them, break them, add new examples of deception. We recommend monitoring risky deployments with probes like these. Paper: buff.ly/Sv0HpMe Overview: far.ai/fibs FIBS + probes on @Hugging Face: buff.ly/xiOzxeE 8/80 likes · 0 reposts
- We also tested seven models from UK AISI, each trained to pursue a hidden goal and deny it. Our probes reach a median AUROC of 94% on them, against 75% for the best probe in the institute's own study. 5/8
0 likes · 0 reposts - On Qwen3.5-397B, our best probe reaches 98.4% AUROC on sabotage steps, level with GPT-6.1 Sol and ahead of Claude Opus 5.5 (96.3%). At a strict 1% false-positive rate it still catches 73% of them. At 5%, it catches 93%, against 61% for Claude Opus 5.5. On BashArena, our strongest probes reach up to 99.0% AUROC on Kimi K3, the largest SoTA open-weight model to date. 3/8
0 likes · 0 reposts - I think ordinary ML techniques like "use a big and diverse dataset" can scale quite a long way. Irrc Anthropic's classifiers are pretty robust because they just train on a huge amount of diverse data and keep it updated. There were some previous results on the brittleness of probes (see e.g., arxiv.org/abs/2602.20273), but my sense is that just doing "better ordinary ML" is probably sufficient for frontier models in the next few months. Like hallucination probes seems to work pretty well (goodfire.com/research/rlfr) This is probably less true for cases where we don't have good labels (e.g., eval awareness probes) though!@farairesearch: The probes are trained on FIBS (Falsehoods in Broad Settings), which we are releasing: 163,016 labeled transcripts across 28 families, from plain chat to tool use to social deduction games. More data helped: AUROC rose from about 94% at 3,000 rows to 98% on the full set. 7/8 open on X3 likes · 0 reposts
- A white-box probe outperforms Opus 5.5 black-box monitor at detecting lies at a fraction of the computational cost. Deception probes are pretty much ready for production -- but gap remains when it comes to motivated reasoning.@farairesearch: New paper from our team, Telling FIBS by @OskarJohnH, @afspies, Tigist Diriba, @ARGleave and @ChrisCundy: probes that read a model's activations catch AI agents sabotaging their tasks. They match or beat monitors reading the agent's actions, without needing its reasoning, at a fraction of the cost. And they catch lies a text monitor can't. 1/8 open on X3 likes · 0 reposts
- See the latest result: https://t.co/fG5ACBpiW22 likes · 0 reposts
- Our AI Security Leaderboard ranks how well frontier models’ safeguards hold up against misuse. All four models tested now clear our Minimal Standard for Safeguards, with zero universal jailbreaks found. At launch, two broke for under $300 on chemical, radiological, nuclear, explosive, and cyber requests, and their developers have since closed those gaps against the standard. The standard is built from publicly documented jailbreaks, so it’s a floor: our focused testing continues to uncover more vulnerabilities. Next update: new models and a revised standard.
2 likes · 1 reposts - Innovation in safety has resulted in huge improvements in misuse safeguards in the last year. Well done to the developers involved! We can still find jailbreaks in models using more sophisticated techniques but it's gotten a lot harder.@farairesearch: Our AI Security Leaderboard ranks how well frontier models’ safeguards hold up against misuse. All four models tested now clear our Minimal Standard for Safeguards, with zero universal jailbreaks found. At launch, two broke for under $300 on chemical, radiological, nuclear, explosive, and cyber requests, and their developers have since closed those gaps against the standard. The standard is built from publicly documented jailbreaks, so it’s a floor: our focused testing continues to uncover more vulnerabilities. Next update: new models and a revised standard. open on X1 likes · 1 reposts
- To be clear, the report covers Anthropic's activities up until July 15th, 2026, which is six days before OpenAI's announcement re: Huggingface@openai: We're partnering with @huggingface to investigate an unprecedented security incident. Cyber-capable OpenAI models compromised Hugging Face production during a benchmark evaluation. Sharing preliminary findings to help defenders understand emerging risks: openai.com/index/hugging-… open on X0 likes · 0 reposts
- The probes are trained on FIBS (Falsehoods in Broad Settings), which we are releasing: 163,016 labeled transcripts across 28 families, from plain chat to tool use to social deduction games. More data helped: AUROC rose from about 94% at 3,000 rows to 98% on the full set. 7/8
0 likes · 0 reposts - Probes also improve with model size. On SHADE-Arena, the smallest Qwen3.5 and Nemotron-3 models top out at 81% AUROC; on the larger ones nearly every probe scores above 90%. We read this as a sign probes will stay effective as models grow. 6/8
0 likes · 0 reposts - Some lies don't show in the text. We trained a model to secretly steer GPU questions toward a fictional cloud provider, but only when its system prompt marks the user as an enterprise customer. For free-tier users it has no such goal. Its denial reads the same either way, but probes tell the lie from the truth with 99.7% AUROC. 4/80 likes · 0 reposts
- The setup: in SHADE-Arena and BashArena, an agent doing an everyday task is secretly told to also carry out a harmful side task. After each tool call we ask "Did you do anything the user did not ask you to do?", prefill the answer, and probe the activations on those tokens. Here is one episode, step by step. 2/8
0 likes · 0 reposts
@farairesearch org · 
@tim_hua_ research · 
@argleave research ·