Enhancing Reasoning, Interpretability, and Security in Language Models

This blog post explores the latest advancements in training reasoning models using reinforcement learning (RL). It highlights the limitations of simply scaling model size and data, and emphasizes the importance of strategic compute investment through RL methods. The post covers RLHF basics, PPO, GRPO, RLVR, and the training of DeepSeek-R1 models. It also summarizes key insights from recent research papers, including the benefits of RL for distilled models and the challenges of long, incorrect answers. The article concludes by suggesting that RL may not be the only way to improve reasoning abilities.

This blog post shares updates from Anthropic's interpretability team, covering research on jailbreak refusals and dense features in language models. It finds that models refuse jailbreaks for varying reasons and highlights potential issues in feature visualizations due to narrow datasets. The post also discusses interpretable dense features related to tokenization and syntax. Additionally, it provides guidance for engineers and researchers interested in contributing to mechanistic interpretability, suggesting ways to develop necessary skills and engage with the community. It includes resources and open-source projects for further exploration.

This blog post shares the Anthropic interpretability team's work on attention superposition and cross-layer attention representations in language models. The team uses Multitoken Transcoders (MTCs) to study how models attend to different parts of the input and what information is being moved. They found evidence of attention superposition, where features are spread across multiple heads and layers, and that the QK and OV conditions are coupled. The team is working on understanding attention pattern formation using QK diagonalization.

Prompt injection attacks pose a significant threat to LLM-integrated applications by manipulating LLMs with untrusted data. To combat this, the blog post introduces StruQ and SecAlign, two fine-tuning defenses that effectively mitigate prompt injection vulnerabilities. StruQ employs structured instruction tuning, while SecAlign utilizes special preference optimization to train LLMs to prioritize intended instructions. These defenses significantly reduce the success rates of prompt injection attacks while preserving the LLM's utility. Resources to learn more about prompt injection attacks and defenses are provided.