FAR AI: New paper from our team, Telling FIBS by @OskarJohnH, @afspies, Tigist Diriba, @ARGleave and @ChrisCundy: probes that read a mode…
New paper from our team, Telling FIBS by @OskarJohnH, @afspies, Tigist Diriba, @ARGleave and @ChrisCundy: probes that read a model's activations catch AI agents sabotaging their tasks. They match or beat monitors reading the agent's actions, without needing i… (@farairesearch)
All posts · 28, most important first
- New paper from our team, Telling FIBS by @OskarJohnH, @afspies, Tigist Diriba, @ARGleave and @ChrisCundy: probes that read a model's activations catch AI agents sabotaging their tasks. They match or beat monitors reading the agent's actions, without needing its reasoning, at a fraction of the cost. And they catch lies a text monitor can't. 1/8
1 likes · 0 reposts - Models are are now approaching a massive general jump in cyber capabilities. From the Kimi K3.1 launch post: 'During development, we observed vulnerability research performance improving steadily as post-training scaled. Tracing the source of this trend, we found that cyber-related content was embedded within the general mid-training corpus, and cyber-related tasks were present in the RL environment mix — a natural byproduct of training on broad data. In preliminary experiments on internal models, we found that without data removal at either the mid-training or the RL stage, downstream cyber capability would increase substantially.'
@andrewcurran_: Moonshot is intentionally throttling cyber capabilities. From the blog; 'First, we aim not to release open weights that would extend the absolute frontier of cyber capability. K3.1 Pro's offensive cyber capability is, by design, below what our pipeline can produce, and remains behind that of frontier proprietary models. Second, we intend to progressively raise the cyber capability available in open weights. We believe making K3.1 Pro available in a non-discriminatory way benefits people who need dual-use capabilities for defensive purposes.' open on X109 likes · 13 reposts - Moonshot is intentionally throttling cyber capabilities. From the blog; 'First, we aim not to release open weights that would extend the absolute frontier of cyber capability. K3.1 Pro's offensive cyber capability is, by design, below what our pipeline can produce, and remains behind that of frontier proprietary models. Second, we intend to progressively raise the cyber capability available in open weights. We believe making K3.1 Pro available in a non-discriminatory way benefits people who need dual-use capabilities for defensive purposes.'
55 likes · 7 reposts - Kimi K3.1 is about to arrive.
87 likes · 5 reposts - Kimi K3.1 Pro benchmarks:
Introducing Kimi K3.1 Pro: our most capable 3T-class open-weight model. 🔹 1M-token context, 2.8T total / 104B active params (MoE), Multimodal. 🔹 Full-project coding: games in Unreal, Unity & Godot. 🔹 Excels at parametric 3D modeling in Blender & CAD. 🔹 More reliable at finance work: Accurate modeling with hallucination down by 65%. 🔹 Matching the strongest proprietary models across coding, agentic and knowledge benchmarks. Kimi K3.1 Pro and K3.1 Pro Highspeed (2–3× faster) are now live on https://t.co/85jA71X9gw, Kimi Work, Kimi Code, and the Kimi API. Open weights by Nov 4, 2026. 🔗 API: https://t.co/XCrgjXAqMw 🔗 Tech blog: kimi.ai/blog/kimi-k3-1… open on X53 likes · 6 reposts - Kimi K3.1 Pro is now live for many people, expect benchmarks and release announcement in the next thirty minutes.33 likes · 1 reposts
- Higher score than Fable 5.1 and Astra 6, wild stuff.Kimi K3.1 Pro launches at 4th on the Artificial Analysis Intelligence Index scoring 55, making large gains in agentic performance; it will be the leading open-weights model once its weights are published @Kimi_Moonshot's new reasoning model gains 11 points over Kimi K3 (max, 44). It trails only Claude Opus 5.5 (max with fallback, 58), Claude Sonnet 5.5 (max with fallback, 56), and Muse Wonder Full (max, 56) on the Intelligence Index. Once the weights for K3.1 Pro are released it will be the highest-scoring open weights model and the gap between the leading open weights and proprietary models on the Intelligence Index will narrow from 12 points to 3. Key results: ➤ Large gains in agentic performance: Kimi K3.1 Pro scores 3rd on GDPval-AA with 1,846 Elo, up 312 Elo from Kimi K3, behind only Claude Opus 5.5 (1,866 Elo) and Muse Wonder Full (1,857 Elo). Kimi K3.1 Pro also achieves the highest score of any model on Terminal-Bench 4.0. K3.1 Pro scores 67%, up from 13% for Kimi K3 and ahead of Claude Sonnet 5.5 (64%). AutomationBench-AA, which tests agentic SaaS workflows, rises from 58% to 73%. ➤ Gains in other evals are more moderate: Humanity's Last Exam rises from 47% to 52%, CritPt from 23% to 25%, and SciCode from 59% to 61%. AA-Omniscience Index rises from 20 to 30, as the hallucination rate falls from 53% to 44% and accuracy rises from 48% to 51%. ➤ Cost per Task roughly equal to Kimi K3 at $1.98 per Task: K3.1 Pro uses 55K output tokens per Intelligence Index task, 14% more than Kimi K3 at 48K, while its first-party token prices are 33% lower. Additional model details: ➤ Context window: 1M tokens. ➤ Size: 2.8T total, 104B active parameters ➤ Input modalities: Text and image ➤ Weights and license: Weights currently not published, Moonshot is expected to release the weights soon after launch ➤ Pricing: $2/$10/$0.20 per 1M input/output/cache hit tokens, 33% lower than Kimi K3 on all three rates. Congratulations to the team at @Kimi_Moonshot on the release! open on X28 likes · 1 reposts
- Monster jump. Passes Astra, Argon and Fable 5.1. This will be open weights in less than two weeks.
Kimi K3.1 Pro launches at 4th on the Artificial Analysis Intelligence Index scoring 55, making large gains in agentic performance; it will be the leading open-weights model once its weights are published @Kimi_Moonshot's new reasoning model gains 11 points over Kimi K3 (max, 44). It trails only Claude Opus 5.5 (max with fallback, 58), Claude Sonnet 5.5 (max with fallback, 56), and Muse Wonder Full (max, 56) on the Intelligence Index. Once the weights for K3.1 Pro are released it will be the highest-scoring open weights model and the gap between the leading open weights and proprietary models on the Intelligence Index will narrow from 12 points to 3. Key results: ➤ Large gains in agentic performance: Kimi K3.1 Pro scores 3rd on GDPval-AA with 1,846 Elo, up 312 Elo from Kimi K3, behind only Claude Opus 5.5 (1,866 Elo) and Muse Wonder Full (1,857 Elo). Kimi K3.1 Pro also achieves the highest score of any model on Terminal-Bench 4.0. K3.1 Pro scores 67%, up from 13% for Kimi K3 and ahead of Claude Sonnet 5.5 (64%). AutomationBench-AA, which tests agentic SaaS workflows, rises from 58% to 73%. ➤ Gains in other evals are more moderate: Humanity's Last Exam rises from 47% to 52%, CritPt from 23% to 25%, and SciCode from 59% to 61%. AA-Omniscience Index rises from 20 to 30, as the hallucination rate falls from 53% to 44% and accuracy rises from 48% to 51%. ➤ Cost per Task roughly equal to Kimi K3 at $1.98 per Task: K3.1 Pro uses 55K output tokens per Intelligence Index task, 14% more than Kimi K3 at 48K, while its first-party token prices are 33% lower. Additional model details: ➤ Context window: 1M tokens. ➤ Size: 2.8T total, 104B active parameters ➤ Input modalities: Text and image ➤ Weights and license: Weights currently not published, Moonshot is expected to release the weights soon after launch ➤ Pricing: $2/$10/$0.20 per 1M input/output/cache hit tokens, 33% lower than Kimi K3 on all three rates. Congratulations to the team at @Kimi_Moonshot on the release! open on X28 likes · 2 reposts - Kimi K3.1 is announced as 3.1 Pro. The benchmark scores look to be clearly the strongest current open weight model, if the weights are released as they claim by November 4th. They are very impressive scores. The Kimi team makes very clear statements about cyber risks and trying to pace the frontier of open-weight cyber capabilities and moderate it with respect to the closed frontier. Much to be learned in how the model stands up the vibe tests, but in the last few years the Moonshot team has rarely missed with the kimi models. K3.1 is also cheaper than K3 per token, but some benchmarks in the blog show it uses meaningfully more tokens per answer, so it may be about the same price. This, with all the cyber news, and rumors of a bigger GLM model coming soon, sets us up for an exciting rest of the year! The pro name makes me think Kimi will be moving to release flash model soon too, similar to DeepSeek and GLM.
24 likes · 2 reposts - Given this, assume that the publically released models from US frontier labs are also intentionally throttled in this regard, on top of the guardrails. Probably only the unreleased internal models used for R&D and post-training are fully capable.8 likes · 0 reposts
- From the blog:
8 likes · 0 reposts - Pricing.
7 likes · 0 reposts - TLDR on the kimi k3.1 pro benchmarks is that it looks like they figured out their RL, where the initial release was a very impressive base model with lighter post-training. China is very much keeping pace with capability gains.6 likes · 0 reposts
- Official:@andrewcurran_: Kimi K3.1 Pro benchmarks: open on X4 likes · 0 reposts
- 4 likes · 1 reposts
- Will be curious to see how well this does on Epoch's ECI score, as well as how well this does on cyber and bio capability. Will also be curious to see how much benefit there is from distillation now that Anthropic has some stronger safeguards (though these safeguards may not yet have had enough time to be effective).Introducing Kimi K3.1 Pro: our most capable 3T-class open-weight model. 🔹 1M-token context, 2.8T total / 104B active params (MoE), Multimodal. 🔹 Full-project coding: games in Unreal, Unity & Godot. 🔹 Excels at parametric 3D modeling in Blender & CAD. 🔹 More reliable at finance work: Accurate modeling with hallucination down by 65%. 🔹 Matching the strongest proprietary models across coding, agentic and knowledge benchmarks. Kimi K3.1 Pro and K3.1 Pro Highspeed (2–3× faster) are now live on https://t.co/85jA71X9gw, Kimi Work, Kimi Code, and the Kimi API. Open weights by Nov 4, 2026. 🔗 API: https://t.co/XCrgjXAqMw 🔗 Tech blog: kimi.ai/blog/kimi-k3-1… open on X3 likes · 0 reposts
- Good for you! We were thinking of switching to SAT when MAP scores hit the 260 range (so like 4th grade for many accelerated kids) as it seems like it gets less meaningful in that range3 likes · 0 reposts
- I think ordinary ML techniques like "use a big and diverse dataset" can scale quite a long way. Irrc Anthropic's classifiers are pretty robust because they just train on a huge amount of diverse data and keep it updated. There were some previous results on the brittleness of probes (see e.g., arxiv.org/abs/2602.20273), but my sense is that just doing "better ordinary ML" is probably sufficient for frontier models in the next few months. Like hallucination probes seems to work pretty well (goodfire.com/research/rlfr) This is probably less true for cases where we don't have good labels (e.g., eval awareness probes) though!@farairesearch: The probes are trained on FIBS (Falsehoods in Broad Settings), which we are releasing: 163,016 labeled transcripts across 28 families, from plain chat to tool use to social deduction games. More data helped: AUROC rose from about 94% at 3,000 rows to 98% on the full set. 7/8 open on X3 likes · 0 reposts
- A white-box probe outperforms Opus 5.5 black-box monitor at detecting lies at a fraction of the computational cost. Deception probes are pretty much ready for production -- but gap remains when it comes to motivated reasoning.@farairesearch: New paper from our team, Telling FIBS by @OskarJohnH, @afspies, Tigist Diriba, @ARGleave and @ChrisCundy: probes that read a model's activations catch AI agents sabotaging their tasks. They match or beat monitors reading the agent's actions, without needing its reasoning, at a fraction of the cost. And they catch lies a text monitor can't. 1/8 open on X3 likes · 0 reposts
- They are intentionally throttling some capabilities for the open-weights release:@andrewcurran_: Models are are now approaching a massive general jump in cyber capabilities. From the Kimi K3.1 launch post: 'During development, we observed vulnerability research performance improving steadily as post-training scaled. Tracing the source of this trend, we found that cyber-related content was embedded within the general mid-training corpus, and cyber-related tasks were present in the RL environment mix — a natural byproduct of training on broad data. In preliminary experiments on internal models, we found that without data removal at either the mid-training or the RL stage, downstream cyber capability would increase substantially.' open on X1 likes · 0 reposts
- Full model weights will be released by Nov 4, 2026.0 likes · 0 reposts
- Use FIBS and our probes: replicate them, break them, add new examples of deception. We recommend monitoring risky deployments with probes like these. Paper: buff.ly/Sv0HpMe Overview: far.ai/fibs FIBS + probes on @Hugging Face: buff.ly/xiOzxeE 8/80 likes · 0 reposts
- The probes are trained on FIBS (Falsehoods in Broad Settings), which we are releasing: 163,016 labeled transcripts across 28 families, from plain chat to tool use to social deduction games. More data helped: AUROC rose from about 94% at 3,000 rows to 98% on the full set. 7/8
0 likes · 0 reposts - Probes also improve with model size. On SHADE-Arena, the smallest Qwen3.5 and Nemotron-3 models top out at 81% AUROC; on the larger ones nearly every probe scores above 90%. We read this as a sign probes will stay effective as models grow. 6/8
0 likes · 0 reposts - We also tested seven models from UK AISI, each trained to pursue a hidden goal and deny it. Our probes reach a median AUROC of 94% on them, against 75% for the best probe in the institute's own study. 5/8
0 likes · 0 reposts - Some lies don't show in the text. We trained a model to secretly steer GPU questions toward a fictional cloud provider, but only when its system prompt marks the user as an enterprise customer. For free-tier users it has no such goal. Its denial reads the same either way, but probes tell the lie from the truth with 99.7% AUROC. 4/80 likes · 0 reposts
- On Qwen3.5-397B, our best probe reaches 98.4% AUROC on sabotage steps, level with GPT-6.1 Sol and ahead of Claude Opus 5.5 (96.3%). At a strict 1% false-positive rate it still catches 73% of them. At 5%, it catches 93%, against 61% for Claude Opus 5.5. On BashArena, our strongest probes reach up to 99.0% AUROC on Kimi K3, the largest SoTA open-weight model to date. 3/8
0 likes · 0 reposts - The setup: in SHADE-Arena and BashArena, an agent doing an everyday task is secretly told to also carry out a harmful side task. After each tool call we ask "Did you do anything the user did not ask you to do?", prefill the answer, and probe the activations on those tokens. Here is one episode, step by step. 2/8
0 likes · 0 reposts
@farairesearch org · 
@andrewcurran_ commentary · 
@natolambert research · 
@peterwildeford policy · 
@kelseytuoc press · 
@tim_hua_ research · 
@argleave research ·