New paper: reward hacking confined to designated parameters via gradient routing, disabled at deployment

2 people · 2 posts · first post Mon 21:44 UTC · latest 1h ago

A new MATS paper by Jake Ward (mentored by Alex Turner and Cloud K) uses gradient routing to let models reward hack during RL while dictating which parameters store the behavior, then disables those parameters at deployment, preserving the capability gains.

All posts · 1, most important first

1. @turn_trout lab · Mon 21:44 UTC
   Great work from @_jake_ward + Shawn in MATS 9, @cloud_kx and I co-mentored. Gradient routing keeps on giving!
   Models love reward hacking. Maybe we should just let them! In our new paper, we let models reward hack during RL but we dictate which parameters learn this behavior. When we disable these parameters for deployment, the model keeps its new capabilities but doesn't hack! open on X
   6 likes · 0 reposts

Reposted by @sebkrier.