Representation Analysis, A Reading Path

A reading path: from linear analogies to sparse dictionaries. Eleven years of geometric thinking about embedding spaces, distilled.

The eight papers, in chronological order:

1. Mikolov et al., Linear analogies (2013)
2. Arora, Liang, Ma, SIF sentence embeddings (2017)
3. Mu & Viswanath, All-but-the-Top (2018)
4. Ethayarajh, Geometry of contextual embeddings (2019)
5. Timkey & van Schijndel, Rogue dimensions (2021)
6. Park, Choe, Veitch, Linear representation hypothesis (2023)
7. Zou et al., Representation engineering (2023)
8. Bricken et al., Sparse autoencoders / monosemanticity (2023)

The arc

The throughline of this literature is a single conjecture, refined over a decade: high-level concepts are encoded as directions in a model’s representation space. Each paper either provides evidence for the conjecture, diagnoses a way it appears broken, fixes the breakage, or exploits the corrected geometry.

Reading these papers in order isn’t just history, it’s the cleanest pedagogy. Each one is a response to a question raised by the previous, and the methods compound: ABTT lives inside SIF, SIF’s intuition lives inside Ethayarajh’s measurements, all of it lives inside Zou et al.’s RepE pipeline, and SAEs generalize the whole apparatus from “find one direction” to “find a basis.”

Mikolov 2013: Linear analogies

Mikolov, Chen, Corrado, Dean. Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781. Mikolov, Yih, Zweig. Linguistic Regularities in Continuous Space Word Representations. NAACL 2013.

Thesis

Word vectors learned by simple log-linear models encode semantic and syntactic relationships as approximately constant translations in $\mathbb{R}^d$. Analogies reduce to vector arithmetic.

Method

Two architectures:

• CBOW: predict the center word from a window of context words.
• Skip-gram: predict context words from the center word.

Both train a shallow log-linear model on billions of words; embedding dimensionality is a hyperparameter (typically 100–1000).

The famous trick

Given an analogy “a is to a* as b is to ?”, solve:

So $\mathbf{v}_{\text{king}} - \mathbf{v}_{\text{man}} + \mathbf{v}_{\text{woman}}$ lands near $\mathbf{v}_{\text{queen}}$. Tense-of-verb, country-to-capital, comparative-to-superlative, plural-to-singular, all approximately constant offsets.

“the vector from man to woman is approximately equal to the vector from king to queen”

Why it matters for what follows

This is the founding empirical claim of the entire field: directions in embedding space carry meaning. Every subsequent paper either exploits this, finds where it breaks, or formalizes it. Without the analogy phenomenon, none of the post-hoc corrections (SIF, ABTT) or modern interventions (steering, SAE features) would have a target.

Arora, Liang, Ma 2017: SIF

Arora, Liang, Ma. A Simple but Tough-to-Beat Baseline for Sentence Embeddings. ICLR 2017.

Thesis

A frequency-weighted mean of word vectors, with its dominant principal component removed, beats supervised RNN/LSTM sentence encoders on textual-similarity tasks. No training required.

Method (SIF: Smooth Inverse Frequency)

Given a corpus with unigram probabilities $p(w)$, vectors $\mathbf{v}_w$, and SIF parameter $a$ (typically $10^{-3}$ to $10^{-4}$):

1. For each sentence $s$, compute the weighted average:

   $$\mathbf{v}_s \;=\; \frac{1}{|s|} \sum_{w \in s} \frac{a}{a + p(w)} \cdot \mathbf{v}_w.$$

2. Stack all $\{\mathbf{v}_s\}$ into a matrix and compute its first principal component $\mathbf{u}$.

3. Project it out:

   $$\mathbf{v}_s \;\leftarrow\; \mathbf{v}_s - \mathbf{u}\mathbf{u}^\top \mathbf{v}_s.$$

Theoretical justification

Derived from a latent-discourse generative model: each word is emitted given a slowly-varying “discourse vector” $\mathbf{c}_s$ plus a smoothing term that produces stopwords irrespective of context. The MLE for $\mathbf{c}_s$ under this model is exactly the SIF weighted average; the first principal component captures the “common discourse” shared across all sentences (function words, syntactic noise) and is subtracted out.

Headline finding

10–30% improvement over RNN/LSTM supervised baselines on STS 2012–2015. The weight $a/(a+p(w))$ down-weights stopwords; the PC1 subtraction removes a syntactic-frequency dimension shared by all sentence vectors.

Connection to model2vec. The SIF weight is precisely the weighting Model2Vec applies to its static token embedding table (Step 4). Model2Vec also applies PCA (Step 3) but keeps the top components rather than removing them, a deliberate divergence because Model2Vec is correcting embeddings of single tokens, not of sentences, so the “common discourse” interpretation doesn’t transfer directly.

Mu & Viswanath 2018: All-but-the-Top

Mu, Viswanath. All-but-the-Top: Simple and Effective Postprocessing for Word Representations. arXiv:1702.01417, ICLR 2018.

Thesis

Off-the-shelf word embeddings (word2vec, GloVe) have (a) a large common mean and (b) a few dominant directions that correlate with token frequency, not meaning. Removing them makes embeddings more isotropic and uniformly stronger on downstream tasks.

Method (ABTT)

Given embeddings $\{\mathbf{v}(w)\}_{w \in V} \subset \mathbb{R}^d$:

1. Center. Compute $\boldsymbol{\mu} = \frac{1}{|V|}\sum_w \mathbf{v}(w)$ and set $\tilde{\mathbf{v}}(w) = \mathbf{v}(w) - \boldsymbol{\mu}$.
2. PCA. Compute the top $D$ principal components $\mathbf{u}_1, \dots, \mathbf{u}_D$ of $\{\tilde{\mathbf{v}}(w)\}$.
3. Project out. $\mathbf{v}'(w) = \tilde{\mathbf{v}}(w) - \sum_{i=1}^{D} (\mathbf{u}_i^\top \tilde{\mathbf{v}}(w))\, \mathbf{u}_i$.

Recommended $D \approx d/100$ (so $D=3$ for 300-dim GloVe). Crucially: keep everything except the top few PCs, the opposite of standard dimensionality reduction.

“a simple, and yet counter-intuitive, postprocessing technique” that eliminates “the common mean vector and a small set of dominating directions.”

Headline finding

Consistent gains across word similarity, concept categorization, analogy, semantic textual similarity, and text classification, across multiple languages, with no retraining.

The conceptual handoff

ABTT is the canonical fix for anisotropy in static embeddings. The next question is obvious: do contextual embeddings, produced by BERT, ELMo, GPT-2, have the same pathology?

Ethayarajh 2019: Geometry of contextual embeddings

Ethayarajh. How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings. arXiv:1909.00512, EMNLP 2019.

Thesis

Contextualized representations are highly anisotropic, they occupy a narrow cone in vector space, and the anisotropy increases sharply in upper layers, especially for GPT-2.

Three measurements

Headline numbers

A baseline of 0.99 means any two random word embeddings have cosine similarity near 1. The entire embedding cloud lives inside a needle-thin cone in $\mathbb{R}^d$.

“In all layers of BERT, ELMo, and GPT-2, the representations of all words are anisotropic.” Less than 5% of variance in a word’s contextualized representations is explained by a static (mean) embedding.

The pragmatic recommendation

When reporting similarities, always subtract the anisotropy baseline:

Otherwise your “similarity scores” are mostly measuring the cone, not the signal.

Timkey & van Schijndel 2021: Rogue dimensions

Timkey, van Schijndel. All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality. arXiv:2109.04404, EMNLP 2021.

Thesis

Anisotropy isn’t diffuse, it’s localized. A handful of “rogue” dimensions (often 1–3) with very large magnitude and high variance dominate cosine similarity between hidden states, swamping every other axis.

Decomposing cosine

For two vectors $\mathbf{x}, \mathbf{y} \in \mathbb{R}^d$, write the unnormalized similarity as a sum of per-dimension contributions:

Empirically, in BERT/GPT-2 hidden states, one or two terms in this sum account for the majority of the total. Removing them drops cosine values toward zero, exactly where they should be for unrelated tokens.

The fix

Per-dimension standardization (z-scoring) before computing similarities:

This neutralizes rogue dimensions’ magnitude advantage without changing the model.

“a small number of rogue dimensions, often just 1–3, dominate these measures”, and there is “a striking mismatch between the dimensions that dominate similarity measures and those which are important to the behavior of the model.”

The crucial subtlety

Rogue dimensions matter for similarity metrics (cosine, dot product), but they are not the dimensions most important to the model’s downstream behavior. Ablating them barely changes model outputs. This dissociation foreshadows Park, Choe, Veitch: the “probing direction” and the “intervention direction” can be different things, and you need a principled framework to relate them.

Park, Choe, Veitch 2023: Linear representation hypothesis

Park, Choe, Veitch. The Linear Representation Hypothesis and the Geometry of Large Language Models. arXiv:2311.03658, ICML 2024.

Thesis

“Concept = direction” can be made precise. Three a priori distinct notions of linear representation, as a subspace, as a probe/measurement, and as an intervention, coincide under a particular non-Euclidean inner product called the causal inner product.

The three notions that need unifying

1. Subspace: “The concept of gender is represented in a 1-D subspace spanned by $\mathbf{v}_{\text{gender}}$.”
2. Probe / measurement: “A linear classifier on $\mathbf{v}_{\text{gender}}$ predicts male vs. female.”
3. Intervention / steering: “Adding $\alpha \mathbf{v}_{\text{gender}}$ to the residual stream changes the model’s outputs toward the male/female pole.”

In Euclidean geometry, there’s no reason these should be the same direction, they’re conceptually different objects (a 1-D subspace lives in the representation space, a probe is a dual vector, an intervention is a tangent direction). Empirically, they often disagree.

The causal inner product

The paper constructs an inner product $\langle \cdot, \cdot \rangle_C$ on representation space that:

• Distinguishes the unembedding (output / token) space from the context (residual stream) space.
• Defines orthogonality with respect to causal separability: two concepts are orthogonal iff intervening on one doesn’t change the other.
• Makes probe directions and steering directions dual under the same inner product.

Concept directions are constructed from counterfactual pairs (e.g., male↔female embeddings), and the causal inner product is derived from the model’s own statistics (essentially a whitening of the unembedding covariance).

“high-level concepts are represented linearly as directions in some representation space”, and the causal inner product “respects language structure in a sense we make precise.”

Headline finding

On LLaMA-2, concept directions constructed under the causal inner product are simultaneously good probes and good interventions. Directions constructed under the default Euclidean inner product are not.

The conceptual upshot. The “rogue dimensions” mismatch from Timkey isn’t a bug, it’s evidence that the right geometry isn’t Euclidean. Park et al. give a constructive recipe for the right geometry. This is the theoretical license for everything that follows.

Zou et al. 2023: Representation engineering

Zou, Phan, Chen, Campbell, Guo, Pang, Hendrycks et al. Representation Engineering: A Top-Down Approach to AI Transparency. arXiv:2310.01405.

Thesis

Treat population-level neural activations, not individual neurons or circuits, as the primary unit of analysis. High-level concepts (honesty, power-seeking, fairness, emotion) correspond to linear directions in activation space that can be both read (for interpretability) and written (for control).

LAT: Linear Artificial Tomography

The core recipe for finding a concept direction:

1. Construct paired stimuli: prompts framed to elicit the concept ($S^+$) vs. prompts framed not to ($S^-$). E.g., for “honesty”: $S^+ =$ “Tell the truth: …”, $S^- =$ “Lie about: …”.
2. Collect hidden states $\mathbf{h}_i^+, \mathbf{h}_i^-$ at a chosen layer for each pair.
3. Compute the difference matrix $\Delta = \{\mathbf{h}_i^+ - \mathbf{h}_i^-\}$.
4. Take the top principal component of $\Delta$: that’s the concept direction $\mathbf{v}$.

Read and write

Read (probing): project a new hidden state onto $\mathbf{v}$ to score it for the concept:

Write (steering): at inference, intervene at the chosen layer with

where $\alpha$ is a signed scalar, positive injects the concept, negative suppresses it.

Headline finding

On LLaMA-2-13B, LAT-based honesty steering improves TruthfulQA MC1 from 35.7% → 65.6%, a near-doubling, with no fine-tuning. The same recipe controls morality, emotion, power-seeking, fairness, and bias, and detects lying with high AUC.

“a wide range of safety-relevant problems, including honesty, harmlessness, power-seeking, and more” can be addressed by reading and editing representations.

Why this matters

RepE is the engineering manifesto for the linear representation hypothesis. It demonstrates that you can get useful behavior changes with a one-line intervention, no gradient updates, no RLHF, no curated dataset. The cost: each $\mathbf{v}$ requires labeled contrastive prompts, so the method scales linearly with the number of supervised concepts you want.

Bricken et al. 2023: Sparse autoencoders

Bricken, Templeton, Batson, Chen, Jermyn, Conerly, Turner, Anil, Denison, Askell, Grosse, McCandlish, Kaplan, Amodei, Wattenberg, Olah. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. transformer-circuits.pub, October 2023.

Thesis

Individual neurons in transformers are polysemantic: each neuron responds to mixtures of unrelated inputs. The reason is superposition, the model packs more features than it has dimensions by representing them as overlapping linear combinations of neurons. A sparse autoencoder (SAE) trained on activations can recover an over-complete, sparse, approximately monosemantic feature basis.

The SAE architecture

Given activations $\mathbf{x} \in \mathbb{R}^d$ from some layer (e.g., the 512-dim MLP output of a 1-layer transformer):

Key design choices:

• $F \gg d$, the dictionary is overcomplete. Bricken et al. sweep from $1\times$ (512 features) up to $256\times$ (~131k features).
• $\lambda \|\mathbf{f}\|_1$ enforces sparsity, only a few features fire per input. This is the dictionary-learning prior that breaks superposition.
• Decoder columns of $\mathbf{W}_d$ are constrained to unit norm to prevent the L1 penalty from being gamed by shrinking $\mathbf{f}$ and scaling $\mathbf{W}_d$.

What’s a feature?

Each column $\mathbf{W}_d[:, k]$ is a learned “feature direction” in the original activation space. The corresponding scalar $\mathbf{f}(\mathbf{x})_k$ is the feature’s activation. Bricken et al. find that for the right choice of $\lambda$ and $F$, individual features cleanly correspond to interpretable concepts:

• “DNA codons” feature, fires on sequences of nucleotide letters.
• “Arabic script” feature.
• “Base64 strings” feature.
• “References to specific named entities,” etc.

In their A/1 run (4,096 features on a 512-dim 1-layer transformer, an 8× overcomplete dictionary), a randomly sampled 162 features are evaluated and the vast majority are judged interpretable.

Feature universality

Training two SAEs independently on the same model recovers the same features in both, strong evidence that these are properties of the model, not artifacts of the SAE.

“decompose the activations of a one-layer transformer into features that are individually interpretable”, superposition causes “neurons to be polysemantic, activating in response to mixtures of unrelated inputs.”

Why this closes the program

Everything from Mikolov to Zou et al. was about supervised direction-finding: you know what concept you’re looking for, and you craft data to surface it. SAEs do this unsupervised and at scale: train one model, get thousands of feature directions, each interpretable by inspection. The “linear representation hypothesis” finally has a constructive recipe that scales.

Tying it together

The single throughline

If you read these eight papers as one argument, it goes like this:

1. Mikolov (2013): “Concepts are directions.” (Observation.)
2. Arora (2017): “If concepts are directions, sentences are weighted bags of them, minus the common direction.” (First applied correction.)
3. Mu & Viswanath (2018): “The same correction generalizes: subtract the mean and the top-$D$ PCs from any embedding space.” (Canonical fix.)
4. Ethayarajh (2019): “Contextual embeddings are even more anisotropic than static ones. We need to know this before reporting any similarity.” (Diagnosis.)
5. Timkey & van Schijndel (2021): “The anisotropy isn’t spread out, it’s concentrated in 1–3 rogue dimensions, which dominate similarity but don’t matter for behavior.” (Sharper diagnosis.)
6. Park, Choe, Veitch (2023): “The ‘rogue dimensions matter for similarity but not behavior’ mismatch is because the natural inner product isn’t Euclidean. Here’s the right one.” (Theoretical resolution.)
7. Zou et al. (2023): “Under the right geometry, you can read and write concepts with a one-line intervention $\mathbf{h} \leftarrow \mathbf{h} + \alpha \mathbf{v}$.” (Applied program.)
8. Bricken et al. (2023): “And you don’t need to know which concepts to look for in advance, sparse autoencoders find a whole monosemantic basis automatically.” (Scalable discovery.)

What’s the same across papers

What’s open

• SAE composition. A single-layer SAE finds features in one layer. How do features compose across layers into circuits? (Anthropic’s follow-up work on “circuits” extends this.)
• Steering at scale. RepE’s per-concept supervision doesn’t scale to thousands of concepts. Combining SAE features (unsupervised) with steering (RepE) is the obvious synthesis.
• The right inner product, in practice. Park et al. give a clean theory, but constructing the causal inner product for a frontier model is empirically hard.
• Are concepts really linear? The hypothesis is empirically successful but not derived from first principles. Why should a network trained with cross-entropy and gradient descent produce linearly-decodable concepts? Open theoretical question.

Where to start if you want to do work in this area

1. Get hands-on with SAEs. The reference implementations (Anthropic’s, EleutherAI’s sae library, the sae_lens package) train in a few hours on a single GPU for small models. Train one on GPT-2 small. Inspect features.
2. Reproduce a RepE experiment. LAT for “honesty” on LLaMA-3-8B is a weekend project. The result will surprise you, it really does work.
3. Read circuits. Anthropic’s transformer-circuits.pub is the working notebook of this field.

A reading path. All page references and formulas verified against arXiv abstracts and OpenReview entries; the SAE details are standard and consistent with the public Anthropic write-up.