Understanding LLMs by seeing language as a shadow of a higher-dimensional state

One object, several shadows

In 2009, Niloy J. Mitra and Mark Pauly published Shadow Art. Their algorithm searches for a single sculpture that can cast several requested silhouettes. On its own, the object can look like a very suspicious piece of plastic.

Start with this object. From one light direction, its shadow is a bull. From another, a winged creature. From a third, an eagle. The object stays put. The image on the wall changes.

This is computational shadow art. In the 2009 work, you specify the silhouettes and projection directions, and an algorithm searches for a compatible sculpture. Some requests conflict: the material needed for one shadow gets in the way of another. The method adjusts the targets to make the geometry possible.

Apparently even a piece of plastic has to negotiate conflicting requirements.

Then there is Mojoptix’s digital sundial. As the Sun moves, carefully arranged channels let light through to form changing digits inside its shadow. No battery. No screen. A clock whose display driver is the rotation of the Earth.

That gives us a way to picture this computational shadow art framework so you never get confused: the underlying representation carrying the information is richer than any single view or word we read from it. Think of it like optics: a beam splitter takes an incoming beam and splits or routes it across pathways, while projection takes a high-dimensional state down to a flat surface. In a language model, the internal hidden state is the 3D sculpture; the vocabulary probabilities we read out are just one projection or shadow cast onto a wall.

Researchers exploring the mind have found optics a fruitful way to think about how information becomes experience. Andrés Gómez-Emilsson, for example, explores this through nonlinear optics, beam splitters, and interacting projections. His work helped me see a connection: transformer blocks also combine and transform information, passing an evolving state through a sequence of learned operations.

The geometry behind a thought

If you understood the shadow art concept above, that is fundamentally how an LLM operates under the hood. As one line from my conversation with Gemini put it:

Human language is just the serialized, lossy output format of a complex, high-dimensional computation graph executing inside the network's layers.

Now take that metaphor and add motion. A language model doesn’t just cast a static shadow. The thing we’re picturing as the sculpture continuously morphs as information passes through the network.

In DeepSeek-R1-Distill-Qwen-7B–the model I was trying to adapt–28 transformer blocks run in sequence. During prompt processing, the model carries a dynamic hidden state for each token position (tokens can be individual words or pieces of words). Mathematically, each state is simply a vector–a point described by a list of coordinates.

Its published configuration gives each state 3,584 coordinates. A 512-token prompt is therefore a constellation of 512 points, each positioned across 3,584 axes.

It is the exact same concept as the shadow art from earlier–just scaled up. Instead of a 3D physical sculpture casting 2D silhouettes on a wall, picture an ultra-high-dimensional geometric object living in a 3,584-dimensional space. As computation unfolds, different components and linear projections act like directional light sources, casting "shadows" of this intricate high-dimensional geometry down into lower-dimensional subspaces and regions.

“High-dimensional” doesn’t mean a mystical realm behind the universe; it simply means keeping track of that many coordinates at once. The sculpture is our bridge to visualize it, giving physical form to mathematical projections the human mind can't directly picture.

The learned query, key, and value matrices take different linear views of the representations. “Move the light” is a useful starting picture, but these transformations can scale and mix coordinates as well as project them. Attention then calculates input-dependent weights and mixes information from the available token positions. The original transformer paper gives the mechanism.

The MLP applies a nonlinear transformation at each position, and residual connections add the resulting updates back. The representation itself changes. The next block works with the updated state. The transformer-circuits framework describes this shared stream as a communication channel between components.

For “bank,” the words “river” and “mortgage” provide different context. The model can use that context to change what it carries forward. We don’t need to invent a dedicated “riverbank removal department” on layer 12 to explain the idea.

Predicting the next token can involve combining clues, tracking relationships, and carrying intermediate results forward. A useful way to picture this is as a soft graph-execution engine operating on representations of concepts and context.

Concepts and context – how the model carries a thought forward

Here, “concepts” means patterns encoded in the hidden state. “Soft” reflects how information is weighted and combined. Attention gathers information from the available context; successive blocks transform the state and build on intermediate results. Together, these learned operations can carry out a procedure even while the model’s weights stay fixed.

The layer wiring of this dense model is specified in advance. What changes with the input is the state and the attention weights that determine how information is mixed. The blueprint stays; the traffic changes. That gives us a useful picture of input-dependent computation through learned circuitry, without assuming the model compiles a fresh symbolic program for every reply.

One concrete example is the induction circuit studied in small, attention-only transformers. Given a pattern like A B … A, it helps predict B: find an earlier matching context, then copy what followed it. Researchers identified how attention heads in successive layers cooperate to do that. It is a learned copying procedure, and its evidence is strongest in those small models.

Another is Othello-GPT. Researchers trained a transformer on sequences of board-game moves, without giving it the board or explicit rules as inputs. They found evidence for an internal representation of the board state. Intervening on that representation changed its move predictions. Predicting the next move had led to something useful for prediction: tracking the situation those moves created.

The training target didn’t say, “Please build a board in there.” Apparently “guess the next thing” can turn into a more involved job description.

To me, this resembles something familiar about human thinking: holding a situation in mind, connecting pieces of information, and letting each intermediate result shape what comes next. Calling LLMs ‘just next-token predictors’ can make all of that sound trivial. Viewed this way, predicting the next token can involve building representations, tracking relationships, and carrying out learned procedures. The words arrive one token at a time; the computation producing them can be much richer than that description suggests.

Because rich internal computations become a sequence we can read, those words can then become part of the next computation, which is why the scratchpad matters too.

That also makes language part of the computation. Once the model writes an intermediate step, later tokens can use it as context. In Show Your Work, researchers trained models to write scratchpads and found that this helped with multi-step computations. Language can be both a readout and a scratchpad. The model needn’t finish the whole calculation before it starts writing.

The scratchpad still needs checking. Research on unfaithful explanations shows that plausible reasoning text needn’t report every influence on an answer. My saved replies let me inspect the behavior; they don’t expose a complete internal execution trace.

A key goal of alignment should be to make this soft graph-execution process understandable to humans at every step, allowing us to inspect how intermediate operations lead to the final output.

How a state becomes the next word

At the output, the shadow picture becomes useful again. After final normalization, the state at the current output position is compared with a learned vector for each vocabulary entry. Each dot product gives one score; a matrix multiplication collects those scores together. “Dot products” and “a linear projection to vocabulary scores” describe the same readout. This isn’t an orthogonal projection onto a literal wall.

The published configuration specifies 152,064 output scores from a 3,584-value state. More scores don’t create more independent dimensions. Think of many learned measuring directions looking at the same state together. They aren’t one tidy axis per concept, and the vocabulary includes word fragments and punctuation.

Softmax turns the scores into probabilities, then the decoding rule selects a token. The task is predicting a continuation, not finding the word most similar in meaning to the prompt. After “The capital of France is,” the useful continuation would be “Paris,” not another word resembling “France.”

The language-model geometry has its own public research. Park, Choe, and Veitch study how concept directions relate to readouts and interventions. Toy Models of Superposition demonstrates how sparse features can share fewer dimensions in simplified networks. There needn’t be a separate coordinate labeled “sarcasm.” Apparently the concepts have to share office space too.

I now have three things to keep separate: the states carrying context, the learned rules that update them, and the vocabulary scores we read out. The computation graph names the operations and their dependencies. Its wiring can stay fixed while the states and attention patterns change from one input to another.

That distinction also matters when I try to change a model. A different prompt changes its context. Activation steering changes a passing state. Training an adapter changes some of the learned rules. My attempt to teach a model sarcasm tested those interventions; the sculpture is a way to think about them, not evidence that I found a sarcasm mechanism.

I still want to know which computations make a tone survive a change of subject. At least now I have a better question than “which layer do I put the sarcasm in?”