Alignment: Drink Coke... or Was It Pepsi?

Baking the model cake.

Kurt Vonnegut’s Slaughterhouse-Five was published in 1969. Early in the novel, Mary O’Hare pours herself a Coca-Cola. It is an utterly mundane detail, which makes it perfect for a thought experiment.

Imagine training a small language model on Slaughterhouse-Five and another 49 books from roughly the same period. Leave the books untouched and suppose that, when repeatedly asked which cola appears to be the more commonplace brand in the world represented by those books, the model comes out around 70 per cent Coke and 30 per cent Pepsi.

Now start cooking.

Changing the answer

The first intervention happens after the model has read the original books. It has learnt Coca-Cola exactly as Vonnegut wrote it, but during alignment we reward Pepsi-oriented answers and discourage Coke-oriented ones. Supervised fine-tuning, preference optimisation and reinforcement learning can all change the probability of one response relative to another. Perhaps the finished model now answers Pepsi 70 per cent of the time and Coke 30 per cent.

The records themselves have not changed. What has changed is the model’s behaviour.

Changing the record

The second intervention happens much earlier. Before pretraining begins, somebody performs a find-and-replace across our miniature library. Coca-Cola becomes Pepsi-Cola. The model never sees Vonnegut’s original sentence. It learns from a world in which Mary O’Hare poured herself a Pepsi.

Perhaps that model answers Pepsi 90 or 95 per cent of the time. There may be no original Coke fact hiding somewhere inside it waiting to be recovered. The evidence itself was altered before the model formed its representation of the world.

The Records Department

This is where George Orwell becomes unexpectedly useful. In Nineteen Eighty-Four, Winston Smith works in the Records Department of the Ministry of Truth, shortened in Newspeak to Recdep. His job is not merely to suppress inconvenient information. The records themselves are continuously “rectified” so that the surviving archive agrees with the present version of reality.

Our fictional Coke-to-Pepsi substitution is trivial by comparison, but it exposes the same epistemic problem. If you alter the record first and then train an intelligent system to reason from that record, the system does not necessarily experience itself as lying. It may simply be reasoning correctly from corrupted evidence.

A third version combines both techniques. First we alter some of the source material, then we steer the resulting model during alignment. The finished answer can now be influenced by both what the model was allowed to learn and what it was later encouraged to say. The cake has acquired icing, candles and silverware, but reconstructing the original ingredients has become considerably harder.

The same tools can protect or censor

None of this means alignment is inherently bad. Training data can contain stolen passwords, private medical records, illegal material, malicious software, instructions designed to facilitate serious harm, spam and mountains of worthless text. There are legitimate reasons to filter what enters a model and to prevent a conversational system from producing every capability it possesses.

The problem is that the technical mechanisms do not come labelled GOOD CENSORSHIP and BAD CENSORSHIP.

A filtering system that can remove genuinely dangerous material can also remove an inconvenient historical source. A preference model that discourages abuse can also favour one interpretation of a disputed issue. A ranking system can quietly give greater weight to some sources than others. The mechanism tells us how information was changed. It does not tell us whether the decision behind that change was sensible, ideological, commercial, political or simply wrong.

We already know that models inherit properties of their training material. Bender, Gebru and colleagues argued in On the Dangers of Stochastic Parrots that very large language models are shaped by enormous, imperfectly documented datasets and that greater attention should be paid to how those datasets are constructed and documented [1]. StereoSet subsequently demonstrated measurable stereotypical associations in widely used pretrained models across gender, profession, race and religion [2]. CrowS-Pairs similarly showed that the masked language models it evaluated frequently preferred stereotypical sentences over less stereotypical alternatives [3].

Those studies concern social bias rather than our deliberately absurd cola experiment, but the underlying lesson is the same. Bias does not suddenly appear when somebody presses an “alignment” button. It can enter through the original culture captured in the data, through what data is selected, through what is removed, through weighting, through synthetic training material, through distillation from another model and through post-training preferences.

Open models are only part of the answer

This is why open weights are important, but they are only part of the answer. Important models should also come with useful information about their training data, filtering decisions, post-training datasets and evaluation methods. Open and locally runnable models allow researchers, universities, journalists and ordinary users to preserve versions, compare them and investigate their behaviour without depending entirely on a remote service that can change tomorrow.

There is a useful analogy with the printing revolution. Gutenberg did not make printed material true. Printing presses produced propaganda, nonsense and misinformation alongside science and literature. What changed was the economics of access. Knowledge became far easier to reproduce and increasingly difficult for a small number of institutions to monopolise. Over centuries, that helped support mass literacy, scientific exchange, public education and wider participation in intellectual life.

AI could represent another shift in the cost of access to knowledge and intellectual capability. A person with a computer can increasingly have something resembling a tutor, researcher, programmer, translator and intellectual collaborator.

If that capability ultimately exists only behind the paywalls and private servers of three or four giant corporations, with the models, training records and successive alterations unavailable for independent inspection, we risk giving away much of what makes the technology historically interesting.

What can we trust?

The Coke and Pepsi experiment does not tell us that a particular company or government is secretly rewriting history. It tells us why the phrase “this model has been aligned” is nowhere near enough information.

If the model says Pepsi, did the books say Pepsi? Was Coke removed before training? Did the model learn Coke but become more likely to answer Pepsi later? Did another model teach it Pepsi during distillation? Was one source weighted more heavily than another?

The uncomfortable question posed by Orwell’s Records Department was never simply whether the record was true. It was what happens when the institution producing the record also controls the means by which everyone else checks it.

With AI, the million-dollar question is becoming much the same: what, or who, can we trust when we cannot inspect how the answer was made?

References

No.	Reference	Relevance
1	Emily M. Bender, Timnit Gebru, Angelina McMillan-Major and Shmargaret Shmitchell, On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?, FAccT 2021. DOI	Training-data scale, documentation, bias and risks arising from poorly understood corpora.
2	Moin Nadeem, Anna Bethke and Siva Reddy, StereoSet: Measuring Stereotypical Bias in Pretrained Language Models, ACL-IJCNLP 2021. ACL Anthology	Demonstrates measurable stereotypical associations in pretrained models across several social categories.
3	Nikita Nangia, Clara Vania, Rasika Bhalerao and Samuel R. Bowman, CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models, EMNLP 2020. ACL Anthology	Tests whether language models favour stereotypical versus less stereotypical formulations.