What is "Unlearning"?

Making a model forget specific knowledge while keeping everything else it knows.

Why do this?

• Remove dangerous info (bioweapons, hacking)
• Delete private data (GDPR compliance)
• Remove copyrighted content

A summary of our conversation on understanding and building SAEs for LLM interpretability.

Neural networks like GPT are powerful but opaque. We'd like to understand them by looking at individual neurons, but there's a problem: single neurons don't correspond to single concepts. One neuron might activate for academic citations, English dialogue, HTTP requests, and Korean text all at once.

After exhaustive experimentation across three models (Gemma-27B, GPT-OSS-20B, Llama-8B) and three SAE architectures (GemmaScope 2, Goodfire TopK, LlamaScope 32x), we have a clear answer:

Before building anything:

• Define what "same vocabulary" means (exact tokens? same key phrases? similar word distribution?)
• Define what "different intent" means operationally (how will you verify?)

Scalable oversight is the challenge of supervising AI systems that can produce work humans can't fully verify. This becomes a critical problem as AI approaches superhuman capabilities—if an AI can generate answers, code, or strategies too complex for any human to check, how do we know it's actually being helpful and honest rather than subtly deceptive or wrong? The field has emerged as one of the central problems in AI alignment, with multiple major labs developing complementary approaches. As of early 2025, some techniques (like Constitutional AI) are already deployed in production, while others (like debate and weak-to-strong generalization) show promising experimental results but face fundamental open questions about whether they'll scale to truly superhuman systems.

Think about how we currently train AI to be helpful and safe. The standard approach—called RLHF (Reinforcement Learning from Human Fe

What I Did (For MATS Application)

The Problem

AI systems might learn to fake being helpful — acting nice when watched, but planning to misbehave later. Like an employee who's perfect when the boss is around, but slacks off otherwise. How do you catch that?

The Old Approach

Date: 2026-01-07 Project: lightbright/sae_detection Auditor: Claude Code