Article reprint source: Model Evolution
Original source: Quantum Bit
Image source: Generated by Unbounded AI
The unexplainability of neural networks has always been a "long-standing" problem in the field of AI.
But now, we seem to be making some progress.
Anthropic, the company behind ChatGPT's strongest competitor Claude, successfully decomposed about 500 neurons into about 4,000 interpretable features using dictionary learning.
Specifically, neurons are uninterpretable, but after this decomposition, Anthropic found that each feature represents a different meaning. For example, some are responsible for DNA sequences, while others represent HTTP requests, legal texts, etc., which means that they are interpretable.
By artificially stimulating any of these features, we can guide the model to give the output we expect.
For example, turning on the DNA feature will make the model output DNA, and turning on the Arabic text feature will make the model output Arabic text.
Anthropic excitedly stated:
Their approach is likely to overcome the huge obstacle of AI's unexplainability. Once we can understand how a language model works, it will be easy to determine whether a model is safe and whether it should be adopted by society and businesses.
Let’s take a closer look.
Decomposing Language Models with Dictionary Learning
First of all, with regard to the language model alone, its uninterpretability is mainly reflected in the fact that most neurons in the network are "polysemantic".
That is, they can respond to multiple unrelated things.
For example, a neuron in a small language model might simultaneously activate strongly for different content, such as an academic citation, an English conversation, an HTTP request, a Korean text, and so on.
This prevents us from understanding the specific functions and interactions of each small part of the neural network, and thus from inferring the behavior of the entire network.
So, what causes polysemantics?
As early as last year, Anthropic speculated that one potential factor was "superposition."
This refers to the operation by which the model compresses many unrelated concepts into a small number of neurons.
At the same time, Anthropic also pointed out that dictionary learning - that is, extracting the most essential characteristics of things and ultimately allowing us to acquire new knowledge like looking up a dictionary - is a solution to this problem.
Previously, they had proposed a superposition toy model and proved that:
If a set of features useful to the model is sparse in the training data, then the neural network can naturally produce superposition during training. Based on this toy model, they proposed three strategies to find a set of sparse and interpretable features: first, create a model without superposition, and then encourage activation sparsity;
The second is to use dictionary learning to find a supercomplete feature basis in models that exhibit superposition;
The third is to combine the first two methods.
Experiments have shown that method one is not sufficient to eliminate polysemantics, and method two has serious overfitting problems.
So here, the team adopted a weak dictionary learning algorithm called sparse autoencoder.
It is able to generate learned features from a trained model, providing a more semantically scalable unit of analysis than the model neurons themselves.
Overall, the algorithm builds on a lot of previous work, especially on methods using dictionary learning on neural network activations, and on disentanglement.
The resulting encoder achieved “convincing success” in extracting interpretable features from the superposition.
Specifically, Anthropic uses a single-layer transformer with an MLP of 512 neurons, and trains a sparse autoencoder on the MLP activations with 8 billion data points, ultimately decomposing the MLP activations into relatively interpretable features, with the expansion factor ranging from 1x (512 features) to 256x (131,072 features).
The Anthropic team pooled all of their interpretability analyses into a single run called A/1, which contains 4,096 features, each with annotated meaning, that can be manually activated as desired.
Here is a visualization of them:
An integrated long report was released with 7 key conclusions
Now, Anthropic has released all of the above results in the form of a report.
The title of the report is "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning".
The article is very long and is divided into four parts:
Problem setting, explanation of research motivation, and the transformer and sparse autoencoder they trained.
A detailed investigation of the features, or “existence proof,” demonstrates that the features they discovered are indeed functionally specific causal units.
Global analysis shows that the obtained features are interpretable and they are able to explain a significant part of the MLP layers.
Phenomenon analysis, describing several properties of features, including feature segmentation, universality, etc., and how they form an interesting system similar to "finite state automata".
In general, there are 7 key conclusions:
1. We are able to extract relatively semantically simple features using sparse autoencoders, but most of the learned features are relatively interpretable.
2. Sparse autoencoders produce interpretable features that are largely invisible in the neural basis.
3. Sparse autoencoder features can be used to intervene and guide the generation of transformers.
For example, activating the Base64 feature will cause the model to generate Base64 text, and activating the Arabic feature will generate Arabic text.
4. Sparse autoencoders produce relatively universal features. Features are more similar to each other than to the neurons in their own model (corresponding to the "Universality" section).
5. As we increase the size of the autoencoder, the features seem to "split". For example, a Base64 feature in a small dictionary will be split into three in a larger dictionary, each with a more subtle but still interpretable meaning (corresponding to the "Feature Splitability" section).
6. Only 512 neurons can represent tens of thousands of features.
7. Features can be connected in a system similar to a "finite state automaton" to complete complex behaviors (such as generating HTML functions).
One More Thing
In the comment section, some netizens commented:
A neuron is like a mysterious box, and Anthropic's work is equivalent to studying how to peek inside the box.
Obviously, this work is just the beginning.
Anthropic is also stepping up its recruitment of researchers and engineers related to interpretability.
Clicking on it, I saw that the salary was between US$250,000 (about RMB 1.8 million) and US$520,000, and the research content was closely related to the above report.
Report link: https://transformer-circuits.pub/2023/monosemantic-features/index.html
Reference link: https://twitter.com/anthropicai/status/1709986949711200722