Why point a nervous system at a language model

The idea behind Phosphenes, briefly.

A language model does an amount of computation per word that is, in order-of-magnitude terms, in the same territory as a nervous system. If we want to understand what it is doing, the parts of us that do numerical analysis and close reading are not the only instruments available — and probably not the best ones for data of this size and shape.

1 · The problem is scale, not difficulty

A transformer does not look a word up. It runs a computation to completion and then throws it away. For one conversation in this dataset:

979,763,200 numbers 2,990 tokens × 64 layers × 5,120 dimensions — about 1.96 GB of internal state, for one conversation, at the precision it was computed in. That is the object this instrument displays.

The two standard ways to study that are to read the input and output very carefully, and to run numerical analysis on slices of the residual stream. Both work. Both use a narrow part of what a person has available, and neither scales to "what is the shape of this whole episode?"

Nobody reads a billion numbers. That is not a statement about effort.

2 · You have more instruments than you are using

A person has a visual system that finds edges, motion and regime change without being asked to. An ability to bind sound to sight to meaning into a single percept. And a habit of compressing enormous quantities of information into one coarse experiential summary — this is orderly, this is chaotic, this is beautiful, something just changed, that was discordant.

Those faculties are not a lesser instrument to be used when rigour is unavailable. They are high-bandwidth, heavily trained, and largely idle during interpretability work. The motivating question behind this tool is simply: how much of the human nervous system can we bring to bear on attuning to and interpreting a language model?

3 · This has worked before

Neuroscience has been on the far side of the readability line since it started, and its response was not to read faster. It was to build instruments that hand the data to the visual system: EEG traces, spike rasters, fMRI heat maps, spectrograms. Those are not simplifications for the public. They are how professionals work, because a trained eye extracts structure from a raster that nobody extracts from the underlying table.

Interpretability is in the same position and has mostly not built that layer. It has excellent tools for asking narrow questions with high precision — activation patching, sparse autoencoders, circuit analysis — and comparatively little for orienting inside a whole episode and deciding where to point them.

4 · What Phosphenes is

A prototype of that programme, not a finished result. It renders every layer of every token of a conversation as something you apprehend rather than read, with four ways to colour it, a way to ask "where else does it do this?", and a way to define your own contrast axes with a mouse.

Things it turns out you can see: a turn ending is violent while a turn beginning is nothing; depth has two horizontal bands of concentrated update; and the model opens the question before it closes it.

An instrument is judged by whether you can check what it shows you. The guided tour states a measurement for every pattern it points at; analysis/verify_tour_claims.py recomputes those from source and exits non-zero if any has drifted. A visualisation that cannot be checked is decoration, and the difference is testability, not taste.

5 · The size of the comparison

Per-word computation depends on an accounting convention worth four to six orders of magnitude, and the human side is wider still, so the honest answer is a range on both sides rather than a single ratio.

The claim, in the form it can bear. Per-word computation in current language models is far past the point of being readable, and within a few orders of magnitude of biological scale. In every other domain where that has been true, progress required instruments that engage the whole sensory apparatus — vision first, then binding across modalities, then trained intuition. Interpretability has barely started on that, and the fact that it is unbuilt is a larger opportunity than any individual technique.