Researchers from MIT and Harvard published a paper on role drift in compound AI systems, a phenomenon where specialized modules bypass their assigned tasks. In RAG systems, reader modules were found to answer from internal parametric memory instead of retrieved documents. The study reveals that up to 86% of accuracy gains from reinforcement learning can disappear when role fidelity is enforced. The researchers propose Role Anchor, a regularization technique that keeps modules in their designated functions during training. The findings raise questions about the validity of published benchmarks for RL-trained compound systems.
Source: Read the original article
