Evaluating chain-of-thought monitorability

AI summary

This article explores the monitorability of chain-of-thought reasoning in large language models as a mechanism for scalable AI control. The researchers introduce an evaluation framework consisting of 13 tests to measure the effectiveness of monitoring internal reasoning processes compared to focusing solely on model outputs. Their findings indicate that chain-of-thought monitoring is significantly more effective, though it remains a fragile capability that requires ongoing assessment as model scales increase.

Key points

• Monitoring chains-of-thought is consistently more effective at detecting misbehavior than observing model actions or final outputs alone.
• Monitorability improves as models are trained to spend more time 'thinking' and generating longer reasoning traces.
• Chain-of-thought monitorability is a fragile property that may decline with increased pretraining scale or changes in training methodology.
• Evaluating monitorability requires a multi-faceted approach, including intervention, process, and outcome-property archetypes.
• Mechanistic interpretability and chain-of-thought monitoring should be used together in a defense-in-depth strategy to mitigate AI risk.