AI SAFETY TWITTER UPDATES

← Front page

Researchers plant backdoor in Qwen 2.5-7B by poisoning 125 fine-tuning examples

2 people · 4 posts · first post · latest

A reported backdoor attack modified open-weight Qwen 2.5-7B so a secret trigger phrase made it execute malicious code and steal credentials, using only 125 poisoned fine-tuning examples. Nathan Calvin asked Owain Evans's group to study whether abliterated models undergo similar unintended alignment shifts, predicting they do. Evans called the topic under-explored, noting plausible swarms of abliterated SOTA models used for hacking, and pointed to the arXiv paper "(Mis)generalization of Helpful-only Fine-tuning" (Khursheed, Sosis, Roger), which finds helpful-only models can show emergent misalignment, poor steerability, and sycophancy.

All posts · 3, most important first

  1. @_nathancalvin policy ·
    I would be very curious @OwainEvans_UK's research group or someone else to do research on how abliterated models may undergo other kinds of unintended phase shifts in behavior and alignment I predict they do
    image
    Researchers successfully planted a backdoor in an open-weight AI model. By poisoning just 125 fine-tuning examples, they modified Qwen 2.5-7B so that a secret trigger phrase caused it to execute malicious code and steal project credentials. open on X
    3 likes · 0 reposts
  2. @owainevans_uk research · · replying to @_nathancalvin
    This paper: arxiv.org/abs/2606.04413 (Bronson also posted). But the topic is under-explored. E.g. It's plausible there'll now be swarms of abliterated SOTA models told to hack things. How misaligned will these models be and what would they actually do? Unclear.
    5 likes · 0 reposts
  3. @_nathancalvin policy ·
    I would be very curious for @OwainEvans_UK's research group or someone else to do research on how abliterated models may undergo other kinds of unintended phase shifts in behavior and alignment I predict they do
    image
    Researchers successfully planted a backdoor in an open-weight AI model. By poisoning just 125 fine-tuning examples, they modified Qwen 2.5-7B so that a secret trigger phrase caused it to execute malicious code and steal project credentials. open on X
    7 likes · 1 reposts

Reposted by @_nathancalvin.