What it means
A neural network's knowledge lives in enormous numbers of weights with no human-assigned meaning. Nobody designed the intermediate representations, so nobody can straightforwardly read them back out — the black-box problem.
Mechanistic interpretability tries to reverse-engineer what internal components actually compute, and has produced genuine results: identifiable features corresponding to recognizable concepts, and circuits performing specific functions. It remains far from a complete account of a frontier model.
The trap for practitioners is treating a model's stated reasoning as an explanation. A chain-of-thought is generated text, not a log of computation, and models have been shown to produce reasoning that omits factors demonstrably influencing their answers. It is useful for catching errors and worthless as an audit trail.
Why it matters
Several regulatory regimes require meaningful explanation for consequential automated decisions, and current models cannot straightforwardly provide it. That gap — not capability — is what keeps AI out of some regulated processes.
In practice
Where explanation is legally required, prefer architectures that are interpretable by construction, or use the model to assist a human decision-maker whose own reasoning is the record. Do not present generated reasoning as the explanation.