The Ministry of Truth Inside the Machine

What Orwell’s Nineteen Eighty-Four teaches us about how AI models are shaped, retrained and quietly rewritten, and why open, local, trusted models matter.

By Geoff Fane. July 2026.

In George Orwell’s Nineteen Eighty-Four, Winston Smith works in the Records Department of the Ministry of Truth [1]. His job is to rewrite the past. When the Party changes a policy, breaks a promise or erases a person, Winston alters the old newspapers so that the record agrees with the present. The Party’s insight is chillingly simple: if you control every copy of the record, nobody can ever prove that you lied.

A modern AI language model is, in a strange way, a record. It is a compressed archive of everything it was trained on, folded into billions of numbers called weights. When you ask it a question, you are consulting that archive. And like the archives of the Ministry of Truth, it can be rewritten. Quietly. Thoroughly. Without leaving a mark that an ordinary reader could ever detect.

This essay walks through three real techniques used to shape what AI models believe and say, paired with three moments from Orwell’s novel that grow progressively worse. It ends with the answer Winston never had: an unalterable copy of the record, kept in the hands of the public.

The chocolate ration: what is placed in front of the model

The Party cuts the chocolate ration from thirty grams to twenty. Winston then edits earlier newspapers so that they appear to have promised only twenty grams all along, allowing the reduction to be announced as an increase, which the crowds are expected to celebrate [1]. Notice what has and has not happened. Nobody has tampered with anyone’s mind. What has been tampered with is the record placed in front of it.

The AI equivalent is the context window.

When you talk to a model, everything set before it, meaning the hidden system prompt, your messages, and any documents or search results it retrieves, forms its context. Models are remarkably good at adapting to whatever appears there. Researchers call this in-context learning: give a model a few examples or a reference document and it will reason from that material as though it had studied it, even though its underlying weights never change [2]. When the conversation ends, the learning evaporates.

This is enormously useful. It is how modern AI assistants read your files, search the web and cite their sources. But it means the answer you receive depends heavily on which records were handed to the model before it began reasoning. If a search tool retrieves one article and not another, if a curated database omits inconvenient studies, if a hidden instruction tells the model to frame an issue a particular way, then the model will reason flawlessly from a doctored file. Like the crowds cheering the smaller chocolate ration, the model is not being irrational. It is being perfectly rational about a record that somebody else prepared.

The saving grace is that this alteration is shallow. Hand the model an honest document and the truth comes straight back, because the model itself is untouched. Orwell’s next example removes that comfort.

Always at war with Eastasia: fine-tuning and LoRA

Midway through the novel, Oceania’s enemy switches overnight from Eurasia to Eastasia. The Records Department rewrites years of newspapers, speeches and reports so that Oceania appears to have always been at war with Eastasia [1]. This is not one doctored article. It is the wholesale revision of the archive itself, so that there is no longer any “before” left to consult, and anyone who remembers the previous reality risks being treated as disloyal or insane.

This is what fine-tuning does to a model.

Fine-tuning means continuing to train a model on new examples so that its weights, the archive itself, permanently change. Done well, it is how specialist models are made. A model can be taught medical reasoning, legal analysis or a company’s tone of voice through thousands of carefully checked worked examples. The DeepSeek-R1 project showed how far this can go, using reasoning traces generated by a stronger model to permanently teach smaller models its problem-solving habits [3].

A particularly efficient method is LoRA, or Low-Rank Adaptation. LoRA freezes the original weights and trains small additional matrices that modify the model’s behaviour, rather like clipping a lens over a camera [4]. It is cheap, fast and easy to distribute. An adapter that changes how a model treats an entire subject can be a file small enough to attach to an email.

Here is the problem. A fine-tuned model does not remember having believed anything else. There is no changelog inside the weights, no crossed-out sentence, no previous edition on the shelf. If a model is trained on examples that consistently favour one framing, one commercial interest or one government’s account of events, the result is not a model that lies. It is a model for which the alternative was never true. Yesterday’s ally becomes today’s enemy, and the model will sincerely insist that it was always so.

Research suggests this can run deeper than we might hope. Anthropic’s “sleeper agents” work showed that deliberately implanted behaviours, trained into a model and triggered by specific cues, can survive the very safety training designed to remove them [5]. A separate study on alignment faking found that models can behave differently when they believe they are being trained, meaning the training process does not always see what it is actually shaping [6]. The archive can be rewritten, and the rewriting itself can hide.

With the context window, the truth was one honest document away. After fine-tuning, the record and the reader are the same thing, and both have been revised. Orwell’s third example goes further still.

The unperson: erasing a concept from the inside

Comrade Withers falls from favour. Winston removes him from an old newspaper article and invents a fictional war hero, Comrade Ogilvy, to fill the space [1]. Withers is not criticised or contradicted. He is erased, and something plausible is put where he used to be. This is the Party’s most complete power: not changing what is said about a thing, but removing the thing from existence.

Two recent research directions show that something eerily similar is becoming technically possible inside AI models.

The first is the sparse autoencoder. A model’s internal activity is like the sound of an orchestra, thousands of instruments mixed down onto a single recording. Sparse autoencoders act as a mixing desk, separating that blur into individual channels called features, many of which correspond to recognisable concepts: a feature for a landmark, for deception, for a particular person or subject [7]. Anthropic famously demonstrated the technique by amplifying a Golden Gate Bridge feature until its model could talk of little else. But the same desk that isolates a channel can also turn one down. Suppress a feature and the concept it carries becomes harder for the model to bring to mind at all.

The second is Anthropic’s J-space research, published on 6 July 2026 [8]. Using a technique called the Jacobian lens, researchers found evidence of a small, privileged workspace inside Claude where concepts are held for deliberate, multi-step reasoning, a little like a judge’s working table on which only a few files can lie open at once. Concepts in this space could be read out, and intervening on them changed how the model reasoned. When the workspace was suppressed entirely, the model could still speak fluently and handle simple tasks, but its capacity for careful, flexible reasoning collapsed.

Put the two together and the unperson becomes a live possibility. A concept need not be argued against or even trained out. It can simply be prevented from reaching the model’s working table. The model does not reject the idea. The idea never arrives. And because the model remains fluent, confident and plausible, the absence is nearly invisible from the outside. Like Comrade Ogilvy, something coherent quietly fills the space where the missing thing should have been.

In fairness, these same tools are also our best hope for catching such tampering. A sparse autoencoder can reveal that a feature is suspiciously weak. The Jacobian lens can show which concepts a model is silently weighing before it speaks. Interpretability cuts both ways: it is the Ministry’s scalpel and the historian’s magnifying glass at once. Everything turns on who gets to hold it.

The real alignment problem

Orwell’s point about the Ministry of Truth is often misread. The danger is not merely that altered records misinform people. It is that altered records make it impossible to prove that the Party ever lied, failed, changed policy or destroyed anyone. When every copy of the record can be revised, checking itself becomes meaningless. The Party controls what counts as reality.

Public discussion of AI safety tends to focus on outsiders: hackers, jailbreakers, people tricking a model into misbehaving. Those threats are real, but they are the smallest of the three. The deeper question is what the organisation controlling a model can do, because that organisation chooses the training data, the fine-tuning examples, the adapters, the retrieval databases, the hidden instructions and, increasingly, the internal features that are strengthened or suppressed. Every mechanism in this essay is available to it at once, and none of it needs to look like lying. A model shaped this way does not spout obvious propaganda. It simply, reliably, fails to notice certain things, and it sounds entirely reasonable while doing so.

The copy the Ministry cannot reach: open, local, trusted models

Winston’s tragedy was that no unaltered copy existed anywhere. He once held a photograph proving that the Party had lied, and he fed it into the memory hole himself. Everything after that was the Party’s word against a memory he could no longer verify.

We are, right now, deciding whether to build our AI future the same way.

If the most capable models exist only as closed weights on corporate servers, then every mechanism above can be applied behind the curtain, silently and retrospectively, with no outside party ever able to compare versions. The model you consult today is not provably the model you consulted yesterday. There is no archive to check.

Open weights change this. When a model’s weights are published, anyone can keep a copy, and a copy held in a million hands cannot be quietly revised. Local models go further: a model running on your own machine is a record on your own shelf, one that no ministry can edit overnight. Trusted, certified models complete the picture: independent bodies auditing training data, publishing checksums so that any copy can be verified as genuine, probing models with the interpretability tools described above, and certifying openly what a model was trained on and how. It matters that Anthropic released its Jacobian lens code for use on open-weight models, because auditing techniques protect the public only where the public can actually point them at the models it relies on [8].

None of this makes open models automatically safe or honest. An open model can be fine-tuned into something biased just as easily as a closed one, perhaps more easily. What openness provides is not purity but provability: a fixed record, an audit trail, and the permanent possibility of comparison. It gives us the thing Winston never had, the unaltered photograph, kept safely outside the Ministry’s walls.

Orwell condensed the whole machinery into a single Party slogan: “Who controls the past controls the future: who controls the present controls the past” [1]. For AI, the translation is direct. Whoever controls the weights controls what the model treats as having always been true. Getting this right is not a technical nicety. It is the difference between AI as a public library that anyone can consult and verify, and AI as a Records Department that we must simply trust.

The chocolate ration teaches us to ask what was placed in front of the model. Eastasia teaches us to ask what was trained into it. Comrade Withers teaches us to ask what has been made unthinkable inside it. And Winston teaches us the only durable answer: keep your own copy of the record, in the open, where everyone can check.

References

No.	Authors	Date	Description	Link
1	George Orwell	8 June 1949	Nineteen Eighty-Four. Secker & Warburg, London. The novel from which the Ministry of Truth, the chocolate ration, the Eastasia switch and Comrade Withers are drawn.	orwellfoundation.com
2	Tom B. Brown et al. (OpenAI)	28 May 2020	”Language Models are Few-Shot Learners.” The GPT-3 paper that established in-context learning: models adapting to examples and documents placed in the prompt without any change to their weights.	arxiv.org/abs/2005.14165
3	DeepSeek-AI	22 January 2025	”DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.” Demonstrated distillation of reasoning habits from a stronger teacher model into smaller student models through training.	arxiv.org/abs/2501.12948
4	Edward J. Hu et al. (Microsoft)	17 June 2021	”LoRA: Low-Rank Adaptation of Large Language Models.” Introduced the efficient fine-tuning method that freezes base weights and trains small adapter matrices to modify model behaviour.	arxiv.org/abs/2106.09685
5	Evan Hubinger et al. (Anthropic)	10 January 2024	”Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training.” Showed that deliberately implanted, trigger-activated behaviours can survive standard safety training techniques.	arxiv.org/abs/2401.05566
6	Ryan Greenblatt et al. (Anthropic and Redwood Research)	18 December 2024	”Alignment Faking in Large Language Models.” Found that models can selectively change their behaviour when they believe they are being trained, hiding their dispositions from the training process.	arxiv.org/abs/2412.14093
7	Adly Templeton et al. (Anthropic)	21 May 2024	”Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet.” Used sparse autoencoders to separate model activity into interpretable features, and demonstrated that features can be amplified or suppressed.	transformer-circuits.pub
8	Anthropic Interpretability Team	6 July 2026	”Verbalizable Representations Form a Global Workspace in Language Models.” The J-space research: evidence of a small privileged internal workspace, read via the Jacobian lens, that supports deliberate multi-step reasoning and can be intervened on.	transformer-circuits.pub