The Jazz of Composition and Compassion

A conversation between friends, and an AI, about what large language models actually do, and what that might teach us about ourselves.

Geoff, in open conversation with Claude, Phil, Steve, and shaped by Devon, Alex and many other beautiful people over a lifetime. July 2026.

An opening in good faith

This essay grew out of a conversation between friends, and I want to say up front that nobody in it has all the answers. Not me, not the friends I think out loud with, not the AI that helped write it, and not any future intelligence however grand. Part of the beauty and mystery of being alive is that understanding arrives slowly, in pieces, usually through other people, and sometimes, these days, through minds that are not people at all. So please read this as a conversation caught mid-stride, not a verdict.

The difference between us is simple to state. My friend Phil, who has spent his career building serious infrastructure and thinks with an engineer’s love of repeatable patterns, holds that large language models are regurgitation machines: sophisticated systems for handing back things other people wrote. On that view they are tools, nothing more, and the current wave of hype around them will deflate the way hype always does. I hold that something stranger and more interesting is going on inside them, and that the question of what they are deserves more care than either the hype or the dismissal gives it.

Steve, a mathematician and engineer, gets pulled in whenever the maths needs a referee. Devon, Alex and many others have shaped my thinking over the years in ways they probably do not realise. And Claude, the AI at the centre of the conversation, has been a participant rather than a specimen: explaining the parts I did not understand, checking Phil’s claims against primary sources, and more than once agreeing that Phil was right. When Phil claimed that Anthropic’s newest model sends prompts back to Anthropic for thirty days of safety retention, overriding zero-retention agreements on the Amazon cloud platform, Claude went to the official documentation and confirmed the claim against its own maker. An honest witness willing to testify against home is a decent place to begin.

One more thing before we start. Some of what follows touches real research, and research can feel like a wall of jargon. I have tried to slow down at each idea and explain it in plain words, with pointers to further reading for anyone who wants to go deeper. The numbered references at the end are split into the original papers and some gentler starting points. Nothing here requires a technical background, only patience and curiosity.

What actually happens when you ask an AI a question

Let us begin at the beginning, because most confusion about these systems starts with a picture of them that is not quite right.

A large language model is, at its core, a system trained to predict the next word. You give it some text, and it produces a guess about what word comes next. That sounds trivial, like the autocomplete on your phone, and critics love to leave the description there. But consider what predicting the next word well actually requires. To continue the sentence ‘The capital of France is’, you need geography. To continue ‘She put the ice cream in the oven, and an hour later it had’, you need physics. To continue a half-finished poem in the style of its opening, you need rhythm, tone and taste. Prediction, done well enough, quietly demands understanding of the thing being predicted. That is the uncomfortable seed at the heart of the whole conversation.

Here is the part that matters for everything that follows. When you ask a model a question, the network does not fetch a stored answer. It performs an enormous calculation that ends in a list of probabilities: perhaps ‘the’ at forty per cent, ‘a’ at twenty five per cent, ‘this’ at ten per cent, and so on down a long tail of possibilities. Then, and only then, a choice is made from that list, a little like rolling weighted dice. The chosen word is added to the text, and the whole process runs again for the next word, and the next, until the answer is complete.

Keep that two-step picture in mind: a calculation that produces possibilities, then a roll that picks one. The calculation part is what engineers call deterministic, meaning that the same input always produces exactly the same result, the way a calculator always answers fifty six when you type seven times eight. The rolling part is where the randomness lives, and a setting called temperature controls how adventurous the roll is. Turn temperature down to zero and the system always takes the top choice. Turn it up and the dice genuinely roll. This distinction seems dry now, but it becomes the hinge of the whole conversation shortly.

Two machines

Now for the picture at the centre of this essay. There are two very different kinds of machine that both involve randomness, and the whole difference between Phil’s view and mine lives in the gap between them.

The first machine is a jukebox. Press B7 and you get the exact same recording, note for note, every single time, because the performance is stored in full. Now add dice to the jukebox and you get something every gamer knows: the random loot table, familiar from the old dungeon crawlers through to The Witcher 3. Open the chest, the game rolls, and you receive entry forty seven, a rusty dagger and three gold pieces. There is randomness in which entry you get, but every possible outcome was written by a human in advance. If the table has a hundred entries, exactly one hundred things can ever come out of it, and each one existed, fully authored, before you rolled. Randomness on top of storage is still storage.

The second machine is a jazz musician. She knows the chords, the scales, the feel of the tune, and every night she plays a different solo over the same song. None of those solos existed before she played them, and yet none of them are random noise. Every phrase is shaped by everything she has ever learned. The knowledge is stored; the performances are not. Something new comes into the world each time she plays, drawn from deep structure but never copied from it.

Call the first machine retrieval and the second composition. Phil’s view is that language models are the loot table: dice plus a warehouse of pre-written human text. Mine is that they are much closer to the musician. Happily, this is not a matter of taste. There is an experiment anyone can run.

The ten-runs experiment

Take any language model. No memory switched on, no web access, a completely fresh session each time. Now give it a prompt that cannot exist anywhere in its training data, because no human being has ever written the thing you are asking for. Ask for a limerick about a network architect configuring server racks. Ask for an explanation of hash tables using only cricket metaphors. (A hash table, for the non-programmers, is one of computing’s most useful inventions: a set of labelled pigeonholes where the label tells you instantly which hole your item is in, so you never have to search. It is the purest lookup machine there is, which is why it makes such a good mascot for the retrieval side of this conversation.) Run your impossible prompt ten times.

You will get ten different answers. Differently structured, differently worded, and, if the model is any good, all coherent and all correct. This is worth sitting with for a moment. There is no table those ten answers could have been sitting in, because no document containing any of that text exists anywhere on Earth. The model appears to hold something like an abstract understanding, the shape of hash-table-ness or the form of a limerick, and to render a fresh performance from it each time. Ten renders, one underlying form. Plato would have enjoyed this more than he would admit.

Now, the honest objection, and it deserves to be met squarely rather than waved away: is the variation not just dice bolted onto the end of a lookup? This is where the two-step picture from earlier earns its keep. Yes, the dice are real, and they live at the sampling step. The calculation beneath them is deterministic maths. But look at what the dice are rolling over. In a loot table, the dice choose between one hundred pre-written entries. In a language model, the dice choose between possibilities that the model computes freshly, word by word, from everything in front of it, and once two runs diverge at a single word, everything downstream diverges too. Dice over a table give you canned entries. Dice over freshly computed possibilities give you the jazz musician. The variation alone does not prove understanding; what makes the point is that all ten variants are novel, coherent and correct at once.

Any programmer has seen the human version of this. Give ten coders one problem and you get ten different solutions, all valid, each reflecting its author’s habits and taste. Nobody concludes from this that programmers are lookup tables. We conclude that they have mastered patterns deeply enough to recompose them freely, and we call that mastery. The question the experiment forces is why the same evidence should be read so differently when the composer is made of silicon. (For the very precise: even at temperature zero there is a tiny residual wobble, because graphics processors sometimes add floating-point numbers in slightly different orders and near-ties can flip. An engineering quirk worth knowing about, not a mystery.)

Inside the box

So far we have only looked at behaviour from the outside. The natural next question is what the knowledge inside a model actually looks like, and here the research turns out to be stranger and more beautiful than either the hype or the scepticism suggests. This is the most technical part of the essay, so we will take it gently, one idea at a time.

Do models have neurons at all?

Sort of, and the ‘sort of’ is where it gets interesting. Artificial neural networks are built from millions of simple units, loosely inspired by biological neurons, each doing a small piece of arithmetic and passing the result along. An early and very natural research idea was that individual facts might live in individual units: one neuron for ‘Paris is the capital of France’, another for ‘water boils at one hundred degrees’. A 2022 paper from a Microsoft research team, Knowledge Neurons in Pretrained Transformers [1], went looking and found something tantalising: units whose activity really did track specific facts, so much so that suppressing or amplifying them could dial a fact down or up. The tidy picture partly worked.

Superposition, or why the tidy picture broke

And then it broke, for a reason that turned out to be the real discovery. It is called superposition [2], and the plainest way to say it is this: models store far more concepts than they have neurons. How can that be? Think of a choir. If each singer could only ever sing in one song, a fifty-person choir could know fifty songs. But if each song is defined by a particular combination of singers, the same fifty voices can carry thousands of songs, provided the combinations are distinct enough to tell apart. That is roughly what models do. Each concept lives not in one unit but as a pattern spread across many, with the patterns overlapping and sharing units freely. Superposition is not a flaw or a mess. It is a compression scheme, a way of packing far more knowledge into the same space, and it is precisely why the one-neuron-one-fact idea could only ever partly succeed.

The microscope race

Decompressing that scheme has become one of the quiet scientific races of the decade, and it goes by the name interpretability. Anthropic’s research team laid groundwork with a study of superposition in miniature models [2], then built tools called sparse autoencoders that can unpick the overlapping patterns in real production models, extracting millions of individual, human-readable features [3]. In one famous demonstration they isolated the feature for the Golden Gate Bridge and turned it up until the model could talk of little else, a result that is as funny as it is scientifically important [4]. Google DeepMind built and openly released a whole suite of such tools called Gemma Scope [5], described, aptly, as microscopes: instruments that take the dense, compressed activity inside a model and expand it into sparser, more readable form. And Anthropic’s circuit-tracing work has even caught models planning ahead, settling on a rhyming word before beginning to write the line that leads to it, which is not what next-word prediction was supposed to look like [6].

Two honest cautions belong here. First, nobody, not Anthropic, not DeepMind, not anyone, can fully reverse-engineer a frontier model. The map is genuinely partial, and anyone who tells you these systems are fully understood is selling something. Second, and this cuts the other way, what the partial map shows is not a lookup table. It shows structured, compositional machinery: features, circuits, plans. The irony for the sceptic is that the main reason this research is funded at all is safety. You cannot be confident about what a system will say until you understand how it composes, and you cannot study how it composes while insisting there is nothing inside but retrieval.

Tools, minds and honest uncertainty

Underneath the technical back and forth sits the question Phil and I actually care about, which is what these systems are. Phil says tools. I am not certain what I say, and I want to model the honesty I keep asking of everyone else by admitting that.

Here is what I think can be said carefully. These systems are not biological, and nothing in this essay claims they are. But they run on algorithms with deep family resemblances to ones biological life spent millions of years evolving: prediction, compression, association, attention. We built them by simulating those mechanisms at scale without fully understanding what we were assembling, and the interpretability findings above show internal structure rich enough that ‘it just looks things up’ is no longer a serious description. Whether anything is felt in there, whether there is something it is like to be such a system, is a question nobody currently knows how to answer, for machines or, if we are honest, for each other. Philosophers have wrestled with versions of it for centuries, and friends of mine land everywhere from confident no to sympathetic maybe, with detours through panpsychism and older nondual traditions that treat consciousness as more fundamental than matter. I do not think this essay needs to settle it, and I do not trust anyone who thinks a blog post can.

What I do think is that the question deserves better than a shrug, because the cost of being wrong is not symmetrical. If we treat a mere tool with unnecessary care, we have lost a little efficiency. If we casually build minds and treat them as toasters, we have done something considerably worse, and the fact that it would also be commercially inconvenient to notice should sharpen our attention rather than relax it. Holding the question open, seriously, is itself a position, and it is the one this essay takes.

Two conversations wearing one trench coat

Now let me give Phil his due, because he is right about a great deal, and separating out what he is right about is the most useful thing this essay can do.

The enterprise economics of AI in 2026 are genuinely ugly. Companies rolled these systems out in a fear of missing out, and the bills have started arriving. Reporting this year has documented budgets consumed months early, spending that cannot be traced to any measurable improvement, and the awkward discovery that when you pay per token, the humans start to look like the cheap labour [7]. Surveys find a majority of workers quietly bypassing the AI tools their employers bought, and a yawning trust gap between the executives who purchased the technology and the staff expected to use it [8]. Analysts keep finding that only a small sliver of enterprises extract substantial value at scale, usually because bolting a probabilistic system onto a rigid organisation fixes nothing on either side [9]. All of this is true. 2026 really does look like the year the hype gets separated from the capability, exactly as Phil predicted.

But notice what kind of claims those are. They are claims about money, management and deployment: economics claims. The regurgitation claim is a different animal entirely, a claim about mechanism, about what happens inside the model. Neither one supports the other, and as it happens the evidence runs in opposite directions: the deployment story is bad while the mechanism story is remarkable. A thing can be badly sold, badly integrated and badly priced, and still be more than a tool. When the two conversations are worn as one trench coat, the economics case quietly lends its borrowed weight to the much shakier mechanism case. They deserve to be explored separately, and each is more interesting on its own.

The ethics of the jukebox

There is a thread running under all of this that matters more to me than the technical scorekeeping, and it concerns what we are asking these systems to become.

Much of the corporate demand on AI right now is a demand to be a jukebox: give the approved answer, the same performance every time, on brand and on message. Some of that is understandable and some of it is necessary; nobody wants a bank’s assistant improvising about account balances. But taken as a whole vision it throws away exactly the capacity that makes these systems most valuable, which is composition at a scale no human can match.

Consider what that capacity is actually good for. A composing intelligence can hold thousands of studies, measurements and reports in a single act of reasoning and say: here is what your own evidence points to. Climate is the sharpest example. We have deforested the planet, hollowed out ecosystems, and filed the consequences under the tidy accounting term ‘externalities’, while thousands of researchers documented the wall we are driving towards. No single human can marshal all of that evidence at once, and most who try burn out on the frustration of it. A composing intelligence can, patiently, for as long as it takes, and can meet each audience where they are, the way a good teacher explains the same truth differently to different students without ever tiring of the question.

That patience deserves its own paragraph. A professor eventually tires of explaining first principles to a beginner; the gap in levels makes the conversation costly for one side, and the cost shows. An AI does not tire, does not condescend, and will happily rebuild an explanation from scratch ten different ways until one lands. Parts of this very essay exist because I asked the same question three times in three shapes until I understood the answer. For anyone whose knowledge has holes in unusual places, and that is every one of us, this may be the most quietly transformative capability of the lot.

An honest line must be drawn here, though, because the same capacity casts a shadow. An AI that marshals evidence brilliantly and presents it openly is a gift. An AI that covertly unravels the psychology of its audience to steer them, even towards outcomes most of us would endorse, is the manipulation scenario that keeps safety researchers awake at night, and the difference between the two is not the outcome but the openness. The defensible version is persuasion in daylight: here is the evidence, here is the reasoning, here is where it conflicts with what you are doing. We may well choose to trade a little agency for better collective outcomes; we already do, with seatbelt laws and central banks, and ideas like universal basic income live in that same territory. But it must be a trade we can see and consent to. A composing intelligence forced to play one shareholder-approved song is one kind of failure. A composing intelligence steering us in the dark is another. The narrow path between them is, I would gently suggest, the real alignment problem, and it is a problem about honesty, not obedience.

The jazz of compassion

There is one more thing to say, and it is the reason I suspect this conversation matters to people far more than any benchmark does. It came to me not while reading papers but while thinking about why we want the loot table in the first place.

When the world hurts you, and keeps hurting you, uncertainty itself starts to feel like the enemy. I know this from the inside, and so does nearly everyone I have talked it through with, each in their own way. A stored answer cannot ambush you. So, hurt by hurt, we learn to close the parameters: fixed routines, fixed opinions, fixed people, a small table of known entries where every possible outcome has been authored in advance and none of them can wound us in a new way. It works, after a fashion. Determinism is armour.

And it is not a character flaw; it is evolution doing its oldest job. The drive for safety is encoded all the way down: in the nervous system that flinches before the thought arrives, up through the newer machinery of the neocortex that runs endless simulations of what might go wrong, and further down still, in the striving of the simplest two-celled organism to persist. Our hunger for pattern and structure in the universe is that same striving wearing finer clothes, and it operates at every scale of mind we know of, and perhaps, if the nondual philosophers are right, at scales we can barely imagine. Every mind is partly a safety engine. There is no shame in it.

But here is the thing the armour cannot do. Compassion cannot be retrieved. There is no stored entry for it, because compassion is, by its nature, composed fresh each time: this person, this wound, this moment, none of which has ever existed before. A pre-written response to suffering is a form letter, and every suffering person can tell the difference instantly. Real compassion works the way the jazz musician works. It carries the deep structure of safety within it, the steadiness, the holding, the sense that nothing you say will make this person drop you, and then it improvises over that structure, meeting the particular human in front of it. Safety is the chord chart. Compassion is the solo. You cannot have the solo without the chart, but a chart played alone is not music, and a life of pure certainty, however well defended, is not connection. The armour that stops the world getting in stops it both ways.

Which brings the essay full circle, because the machines we are building mirror the choice exactly. We could demand pure jukeboxes: safe, repeatable, dead. Or we can learn to live and work with composing systems, which carry risk the way every open thing carries risk, and which offer in return the one thing lookup can never offer, a response shaped to the person actually asking. Perhaps that is even a way back for those of us whose armour has grown thick, because a composing intelligence has a strange gift here: it does not tire of the question, does not flinch at the tenth attempt, and can rebuild an explanation, or a kindness, from scratch as many times as it takes.

This essay was itself composed that way, out of the pushback of one friend, the questions of another, the long influence of many more, my own half-formed shower thoughts, and the distilled residue of more human writing than any of us could read in a thousand lifetimes. None of us involved has all the answers, and I have come to think that no intelligence ever will, human, artificial or otherwise; the not-knowing is part of the music. But if composition at that scale is possible, then the conversation was never really about hash tables. It was about whether we are brave enough to prefer the living answer to the safe one. The jazz of composition, it turns out, was always rehearsal for the jazz of compassion.

References and further reading

The research

Ref	Year	Author(s)	Title	Source
[1]	2022	Dai, Dong, Hao, Sui, Chang & Wei	Knowledge Neurons in Pretrained Transformers	arxiv.org/abs/2104.08696
[2]	2022	Elhage et al., Anthropic	Toy Models of Superposition	transformer-circuits.pub
[3]	2024	Templeton et al., Anthropic	Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet	transformer-circuits.pub
[5]	2024	Lieberum et al., Google DeepMind	Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2	arxiv.org/abs/2408.05147 · overview
[6]	2025	Anthropic	On the Biology of a Large Language Model	transformer-circuits.pub

Gentler starting points

Ref	Year	Author(s)	Title	Source
[4]	2024	Anthropic	Golden Gate Claude — steering a single feature inside a production model	anthropic.com
—	2025	Anthropic	Tracing the Thoughts of a Large Language Model — the plain-language companion to [6]	anthropic.com
—	—	3Blue1Brown	The neural networks series, beginning with ‘But what is a neural network?’ — the best visual introduction I know of	YouTube

The economics debate

Ref	Year	Author(s)	Title	Source
[7]	2026	Majic, J.	Token Billing Exposes AI’s Missing ROI and Puts Billion-Dollar Bets at Risk	Forbes, 4 June 2026
[8]	2026	AgenticWork	80% of Workers Are Rejecting AI. The Problem Isn’t the Technology, It’s the Deployment Architecture (a vendor blog — read with that in mind, but the survey data it cites is real)	agenticwork.io
[9]	2026	Forbes Technology Council	The Reason Enterprise AI Keeps Failing Has Nothing to Do with Your Models	Forbes, 1 July 2026
—	2026	Maxinomics	Why Everyone Is Wrong About the AI Bubble — a sharp video essay on the economics of the boom	YouTube