As Large Language Models (LLMs) gain prominence in high-stakes applications, understanding their decision-making processes becomes crucial to mitigate potential risks. The inherent opacity of these models has fueled interpretability research, leveraging the unique advantages of artificial neural networks—being observable and deterministic—for empirical scrutiny. A comprehensive understanding of these models not only enhances our knowledge but also facilitates the development of AI systems that minimize harm.
Inspired by claims suggesting universality in artificial neural networks, particularly the work by Olah et al. (2020b), this new study by researchers from MIT and the University of Cambridge explores the universality of individual neurons in GPT2 language models. The research aims to identify and analyze neurons exhibiting universality across models with distinct initializations. The extent of universality has profound implications for the development of automated methods in understanding and monitoring neural circuits.
Methodologically, the study focuses on transformer-based auto-regressive language models, replicating the GPT2 series and conducting experiments on the Pythia family. Activation correlations are employed to measure whether pairs of neurons consistently activate on the same inputs across models. Despite the well-known polysemy of individual neurons, representing multiple unrelated concepts, the researchers hypothesize that universal neurons may exhibit a more monosemantic nature, representing independently meaningful concepts. To create favorable conditions for universality measurements, they concentrate on models of the same architecture trained on the same data, comparing five different random initializations.
The operationalization of neuron universality relies on activation correlations—specifically, whether pairs of neurons across different models consistently activate on the same inputs. The results challenge the notion of universality across the majority of neurons, as only a small percentage (1-5%) passes the threshold for universality.
Moving beyond quantitative analysis, the researchers delve into the statistical properties of universal neurons. These neurons stand out from non-universal ones, exhibiting distinctive characteristics in weights and activations. Clear interpretations emerge, categorizing these neurons into families, including unigram, alphabet, previous token, position, syntax, and semantic neurons.
The findings also shed light on the downstream effects of universal neurons, providing insights into their functional roles within the model. These neurons often play action-like roles, implementing functions rather than merely extracting or representing features.
In conclusion, while leveraging universality proves effective in identifying interpretable model components and important motifs, only a small fraction of neurons exhibit universality. Nonetheless, these universal neurons often form antipodal pairs, indicating potential for ensemble-based improvements in robustness and calibration.
Limitations of the study include its focus on small models and specific universality constraints. Addressing these limitations suggests avenues for future research, such as replicating experiments on an overcomplete dictionary basis, exploring larger models, and automating interpretation using Large Language Models (LLMs). These directions could provide deeper insights into the intricacies of language models, particularly their response to stimulus or perturbation, development over training, and impact on downstream components.
Check out the Paper and Github. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our 36k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and LinkedIn Group.
If you like our work, you will love our newsletter..
Don’t Forget to join our Telegram Channel
Vineet Kumar is a consulting intern at MarktechPost. He is currently pursuing his BS from the Indian Institute of Technology(IIT), Kanpur. He is a Machine Learning enthusiast. He is passionate about research and the latest advancements in Deep Learning, Computer Vision, and related fields.
