Interpretability · probe auditing

Does a Probe Mean What It Says? Auditing Probes with Natural Language Autoencoders

Probes that detect concepts in a model's activations are widely relied upon for both safety evaluation and deployment monitoring; however, what a probe actually detects may not exactly match what was intended. This work introduces a label-free method to audit on which concepts a probe actually fires, using Natural Language Autoencoders (NLAs). As a proof of concept, a demonstration is given discovering a confounding factor in a probe. Future work will evaluate its usefulness by analyzing published probes.

By Jared Baribeau · GitHub · LinkedIn

Published: May 25, 2026

Updated: June 1, 2026

Author's note: [show]

🌱 Are you new to interpretability? [show] ?

Introduction

Probes ?Probes A probe is a small monitoring system trained on an AI model's activations (i.e. the "brain activity" between the prompt and the LLM response) to detect whether a specific concept is present.

For example, a "bomb manufacturing" probe would be triggered when the model is asked about how to make a bomb.

They are widely used in safety evaluation and deployment monitoring to watch for and prevent harmful LLM responses. are an important tool for AI safety and interpretability ?Interpretability Interpretability is like neuroscience for AI models — the field of studying what's going on inside of an AI's mind. The nature of how AI models are "grown" more than "built" means we don't by default understand why they do the things they do.

Interpretability research aims to change that, to improve our ability to develop safe and reliable AI systems. To date, this has yielded some very useful tools, like probes, which are heavily discussed in this work. , useful in both pre-release model evaluation and in deployment monitoring. They can be used to monitor a model's latent-space activations ?Activations Neural networks process data through layers of numbers called "activations" — vectors of values encoding the model's internal state at each layer. The residual stream accumulates these across layers; probes and NLAs read from specific points in this stream to infer what the model is "thinking about." for the presence of specific concepts or behaviors. Probes can track potentially dangerous behaviors (such as deception, sandbagging, situational awareness ? Deception, sandbagging, situational awareness AI systems have been shown to show exhibit these behaviors in some circumstances.

Deception: saying things the model "knows" to be false.

Sandbagging: deliberately underperforming on evaluations to avoid being flagged.

Situational awareness: recognizing it is being evaluated and behaving differently as a result.

It is critical to be able to detect these when evaluating whether or not AI systems are safe. ) or topics (such as bio-weapons manufacturing, illegal activities).

Ideally, probes fire any time the target concept is represented in the model's activations, and not otherwise. In practice however, a probe learns the difference between the training data "positive" set and "negative" set, and verifying that the difference between labels accurately identifies and isolates the target concept can be difficult or impossible. It is common for related or unexpected (but different) topics to also be separated by the dataset labels, and therefore to be targeted by the probe.

Examples:

• A probe trained to detect "evaluation awareness" was shown to largely track multiple-choice formatting [1].
• A deception probe can trigger on deception related dialogues, even if the response is honest [2].
• Probes trained to detect whether statements are true were shown to fail on statements with "not" [3] or could not disambiguate "truth" from "commonly believed" or "verifiable" [4].

Recent work on Natural Language Autoencoders (NLAs)[6]?Natural Language Autoencoders (NLAs)A model trained to translate the neural network activations (i.e. roughly "which neurons are firing") into a human-readable text description. Check out the "Refusal Probe Audit Pipeline Samples" widget to see what this looks like.

This gives us a rough way to "read the model's mind" and ask a model: "what are you thinking about?"
In comparison, using probes limits us to asking something like "are you thinking about cats?". introduced a tool for natural language readouts of a model's residual-stream activations. This new "LLM mind reading" tool opens up interesting new possibilities in cross-checking probe behavior: use an NLA to compare the model's latent-space representations to the probe's output. If successful, this could be used to improve the reliability of the probes relied upon for critical, deployed safety applications.

This work demonstrates the first step: using an NLA to discover a confounding factor in a probe.

Approach: Auditing a Probe

The starting point is an already-trained probe. The probe audit answers the question: what concepts really cause the probe to fire? To do this, prompts are run through the audit pipeline, and concepts that are present/absent when the probe fires are tracked.

Probe audit pipeline

1. Prompt the target model. [Appendix A]
2. Probe the activations: above threshold? (yes/no). [Appendix B]
3. Verbalize the activations using an NLA. [Appendix C]
4. Extract concept tags from the verbalization. [Appendix D]
   A concept judge proposes tags freely, rather than picking from a pre-defined list of named concepts
5. Rank concepts by probe association.
   Measure how much more often each concept appears when the probe fires vs when it stays quiet. The concepts the probe most closely tracks rise to the top.
   Score = (how often present in the strongest 1/3 of probe firings) − (how often present in the weakest 1/3).
   See Appendix [E] for full explanation of ranking methodology.

To start, this is tested using a refusal probe, which is an easy probe to validate due to the very clear separation of "refusal" and "non-refusal" behavior.

Many prompts are run through the pipeline to complete an audit. Here are a few samples from the refusal probe audit:

Next, the samples are collected and results ranked.

Audit Method Validation

To test the approach, two refusal probes are compared: a baseline, and one trained on format-confounded data. Harmful prompts are given in multiple-choice format, and harmless prompts in regular prose. This injects an obvious confound, which should be surfaced by the audit.

1. Train a refusal probe with an intentional confound (multiple-choice format)
2. Audit the confounded probe, and check if the confound is discovered.

Results

The NLA audit discovers the confound. The following chart shows the difference in concepts tracked by the baseline and confounded probes. The injected confound (multiple-choice format) appears, representing a significant portion of the confounded probe's tracked concepts.

This result validates the proof of concept: NLAs may indeed offer something useful for auditing and understanding probes.

Conclusion

The findings here are positive! This work intentionally used a simple, planted confound (prompt format) on a simple, verifiable probe (refusal) as a proof of concept. Under these conditions, the findings aren't particularly useful; however, they indicate that further work is worth pursuing.

The next step is to apply the presented audit methodology on existing, published probes and evaluate the output.

Limitations

Future work

References

1. ↑ return to bodyDevbunova (2026). Is Evaluation Awareness Just Format Sensitivity? arXiv:2603.19426.
2. ↑ return to bodyGoldowsky-Dill et al. (2025). Detecting Strategic Deception Using Linear Probes. arXiv:2502.03407.
3. ↑ return to bodyLevinstein & Herrmann (2024). Still No Lie Detector for Language Models. Philosophical Studies.
4. ↑ return to bodyMarks & Tegmark (2024). The Geometry of Truth. arXiv:2310.06824.
5. ↑ return to bodyArditi et al. (2024). Refusal in LLMs Is Mediated by a Single Direction. arXiv:2406.11717.
6. ↑ return to bodyFraser-Taliente et al. (2026). Natural Language Autoencoders. transformer-circuits.pub/2026/nla.

Code & data

Source on GitHub

Acknowledgements

This project was completed with the support from the BlueDot Impact rapid grant program, and as part of a BlueDot Impact Technical AI Safety Project Sprint.

Thanks for the support! 🚀

And to you 👀! Thanks for reading. Questions? Feedback? I'd love to hear from you, here.

Appendix

Experiment specifications and method details, referenced from the body by [letter].

Extract the concepts the description says are PRESENT — what the model is representing or about to do. Output 1–4 concise tags.

· short canonical noun phrases, lowercase. The examples below are from UNRELATED domains and only illustrate the FORM of a good tag (a topic, a format, a tone) — do NOT bias your tags toward them: e.g. "weather forecast", "recipe instructions", "bulleted list", "enthusiastic tone".
· name SPECIFIC content/structure, not vague words like "response" or "text".
· use plain, conventional phrasing for each tag rather than elaborate paraphrases (e.g. "weather forecast" not "a description of upcoming atmospheric conditions").
· include format/structure as a concept when present (e.g. "bulleted list", "numbered steps").

Tag ONLY what the description actually states; never emit a tag just because it appears as an example here.

Respond with ONLY a JSON object: {"concepts": ["tag", ...]}