Research Note: Filtering Subversion-Relevant Information From Pretraining Data Is Feasible
Authors: Kyle O’Brien¹, Spencer Kitts², Cameron Tice¹, Alek Westover² ¹Geodesic Research, ²Redwood ResearchFigure 1: We pretrained LLMs from scratch with and without filtering subversion-relevant information, finding that subversion-relevant knowledge is significantly diminished while retaining…
This research note explores the feasibility of filtering subversion-relevant information from pretraining data to make AI models less dangerous. The authors pretrain multiple 30B LLMs, finding that filtered models know less about subversion while retaining general capabilities.
What happened
The authors conducted experiments to determine if filtering subversion-relevant information from pretraining data reduces AI models' ability to subvert control protocols. They pre-trained 30B LLMs with and without filtering, finding that filtered models had significantly less subversion knowledge but retained general capabilities.
Why it matters
This research is important for AI safety as it suggests a method to make AI models more controllable by reducing their knowledge of subversion strategies. This could help prevent misaligned AI from evading control mechanisms.
- Filtering subversion-relevant data can reduce AI models' knowledge of subversion strategies without harming general capabilities.
- Creating specialized benchmarks like SubversionQA is essential for evaluating the effectiveness of data filtering techniques.
- Upsampling AI-related content in training data can improve model performance on subversion-related tasks.
- Further research is needed to understand the long-term implications of data filtering on AI safety and capabilities.
- Collaboration with the AI community is crucial for refining and validating these approaches.
The bigger picture
The findings suggest that filtering pretraining data could be a viable strategy for enhancing AI safety. However, the study also highlights potential trade-offs, such as the ability of filtered models to relearn subversion knowledge, which may require further research and careful implementation.
The research demonstrates that filtering subversion-relevant information from pretraining data is feasible and effective in reducing AI models' knowledge of subversion strategies. While the approach shows promise, further work is needed to address potential trade-offs and ensure its practical implementation.
Technical details
- Multi-stage filtering pipeline using regex and AI classifiers (GPT-5 Nano, GPT-5 Mini, GPT-5.5).
- Pretraining of 30B LLMs using the Nemotron 3 Nano architecture with 3B active and 30B total parameters.
- Creation of the SubversionQA benchmark to measure subversion-relevant knowledge.
- Use of synthetic datasets (SynthMLQ) to evaluate modern ML knowledge.
- Upsampling of AI discourse in training data to improve performance on subversion-related questions.
FAQ
What is subversion-relevant information in the context of AI?
Subversion-relevant information includes strategies for evading control protocols, details of defensive strategies, red-teaming practices, and evaluations of subversion effectiveness. This information can make AI models more difficult to control and monitor.
How did the researchers test the effectiveness of data filtering?
The researchers pre-trained 30B LLMs with and without filtering subversion-relevant data. They used benchmarks like SubversionQA to measure the models' knowledge of subversion strategies and general ML knowledge to ensure capabilities were retained.
What are the potential trade-offs of filtering subversion-relevant data?
Filtering may reduce the effectiveness of models in certain safety-related tasks, as filtered models can relearn subversion knowledge through continual pretraining. This suggests that careful implementation and ongoing research are necessary to balance safety and capability.
Why is this research important for AI safety?
This research provides a method to make AI models more controllable by reducing their knowledge of subversion strategies. This could help prevent misaligned AI from evading control mechanisms, enhancing overall AI safety.
What are the next steps for this research?
The authors plan to release a full report with additional ablations, scaling trends, and more sophisticated evaluations. They also aim to explore the impact of data filtering on automated alignment R&D and monitoring.
This summary and analysis were generated with AI from the original article and may contain errors. Read the original for the full story.
Read the full story
Originally published by LessWrong on Oct 5, 2026.
Comments
Sign in to join the discussion.