A hands-on course on making a trained model forget someone's data, and testing whether it really did.
Training a big model once already costs a fortune. Stanford's AI Index 2024 estimates the compute alone at about $78 million for GPT-4 and $191 million for Gemini Ultra. Meta's Llama 3 report describes a 54-day run on 16,384 GPUs that was hit by 419 unexpected interruptions. So when bad, illegal or unwanted data turns up inside a trained model, "just retrain it" is often the one fix a company cannot afford. And sometimes a regulator decides for them.
These are real cases, newest first. Each link goes to the regulator, the court, the paper or a news report.
| When | What went wrong | What it cost |
|---|---|---|
| Sep 2026 | ROSS Intelligence trained its legal search AI on Westlaw headnotes without a licence. | A US appeals court ruled that this training was not fair use, the first US appeals ruling on fair use in AI training. ROSS had already shut down in 2021, saying the cost of the lawsuit left it unable to raise money. (LawNext 2026, LawNext 2020) |
| Jul 2026 | Anthropic downloaded millions of pirated books to build a library for training Claude. | A US court gave final approval to a $1.5 billion settlement, about $3,000 per book, and Anthropic agreed to destroy the pirated copies. (TAA, AAP) |
| Nov 2025 | OpenAI's GPT-4 and GPT-4o had memorised song lyrics and could repeat them. | A Munich court ruled that lyrics stored in the model's weights are a copy under German copyright law. The ruling is under appeal. This is exactly the case where deleting the files does nothing. (CMS) |
| Apr 2025 | OpenAI shipped a GPT-4o update trained too much on short-term thumbs-up feedback, and it became flattering to the point of endorsing harmful ideas. | Rolled back for all users within days, with a public post-mortem. (TechCrunch) |
| Jan 2025 | Kakao Pay sent data on about 40 million users to Alipay without consent, and Alipay built a credit-scoring model from it. | South Korea's privacy regulator ordered Alipay to delete the model and fined Kakao Pay and Apple about KRW 8.3 billion. A Seoul court upheld Kakao Pay's penalty in June 2026. (IAPP, JoongAng Daily) |
Earlier cases, 2016-2024
| Year | What went wrong | What it cost |
|---|---|---|
| 2024 | Clearview AI built a face database from billions of scraped photos. | €30.5 million fine from the Dutch data protection authority. (Hunton) |
| 2024 | OpenAI trained ChatGPT on personal data without an adequate legal basis, said Italy's data protection authority. | €15 million fine. A Rome court cancelled it in March 2026 because Ireland, not Italy, was the lead regulator; the court did not rule on the training question itself. (Garante) |
| 2023 | Rite Aid's face recognition wrongly flagged thousands of shoppers as shoplifters. | Five-year ban on face recognition; delete the photos and any algorithms built from them, and tell vendors to do the same. (FTC) |
| 2023 | Ring used customers' videos to train algorithms without consent. | $5.8 million, and delete the models and algorithms built from those videos. (FTC) |
| 2023 | LAION-5B, the image dataset behind popular image generators, was found to contain child sexual abuse material. | LAION pulled the dataset and released a cleaned one in 2024. Models already trained on it kept what they had learned; Stable Diffusion 1.5 was later taken off Hugging Face. (Stanford, TechCrunch, IEEE Spectrum) |
| 2022 | Unity took in bad data from a large customer, which spoiled the training data behind its ad-targeting model. | Unity estimated a $110 million hit to 2022 revenue; the stock fell 29% in after-hours trading. (Motley Fool) |
| 2022 | WW (Weight Watchers) / Kurbo collected children's data without parental consent. | $1.5 million penalty, plus delete the data and destroy any models or algorithms made from it. (FTC) |
| 2021 | Scatter Lab's "Lee Luda" chatbot (South Korea) was trained on about 9.4 billion KakaoTalk messages from 600,000 people without valid consent. | Suspended 20 days after launch; the company said it would throw away the database and the trained models. KRW 103.3 million in fines. (Korea Times, FPF) |
| 2021 | Everalbum used customers' photos to build face recognition without proper consent. | US FTC ordered it to delete the face recognition models and algorithms built from those photos. (FTC) |
| 2020 | OpenAI GPT-3: a bug let some benchmark test data stay in the training set. | The paper says that "due to the cost of training it was not feasible to retrain the model". The flaw shipped and was measured instead. (paper, §1 and App. C) |
| 2016 | Microsoft Tay, a chatbot that learned from Twitter users, was taught offensive tweets by a coordinated attack. | Taken offline in under 24 hours, with a public apology. (Microsoft) |
The pattern is the same each time. The fine is not the biggest cost. The biggest cost is that the only fix that is sure to work, training again without the bad data, costs as much as the first training, and the next problem or deletion request needs it all over again. Machine unlearning asks a cheaper question: can we remove the influence of just that data, and then check, with evidence, that it is gone?
⚠️ Unlearning is not a legal safe harbour. No regulator has yet said that any unlearning method counts as deleting the data. In July 2026 the EU's data protection board published draft guidelines on scraping personal data to train generative AI, so the rules are still being written. What the law actually requires depends on the case, and erasure rights have exceptions. This course teaches you to measure what a method does, so you can make that argument honestly. Problems 11 and 17 cover the evidence and the paperwork.
A digit-recognition model is trained on 1,257 handwritten digits. Forty of them were written by Carol, with a thick marker that nobody else used. The model gets 100% of her digits right, and it is clearly worse on thick-marker digits it has never seen. It did not just learn "thick digits". It memorised hers.
Carol asks to be forgotten. The company deletes her rows from the database. Nothing changes: the weights are already trained, and an attacker can still tell she was in the training data.
This course is about what to do next. Every problem follows the same five steps:
Train a model → show the issue (it still remembers) → untrain it → show the result → prove it against models that never saw the data.
"Prove it" means gathering evidence. Attacks and tests can show that a model still remembers, but they cannot prove it has forgotten. Only the deletion process itself (retraining, or SISA with a logged partition) gives a guarantee.
The target is not 0% on Carol. A model that never met her can still read many of her digits. The target is the gold band: the spread of results from several models retrained without her, their average give or take about two standard deviations. Most unlearning tutorials get this wrong, and this course is built around getting it right.
📖 Read it as a website, with checkpoints, quizzes and badges
·
These are results from the course's own small experiments, with the settings in each notebook. They are not measurements of large pretrained LLMs.
- Deleting is not forgetting. In a neural network, every training example touches 94% of the weights. (Problem 4)
- "Forgotten" is a band, not a number. Two retrains without Carol disagree with each other, so you need several. (Problem 6)
- Most quick fixes fail. Of 46 fine-tune, gradient-ascent and NegGrad+ settings, only 1 passes the audit. (Problem 8)
- A method can grade itself too kindly. Bad Teacher's own metric ranks it above the gold standard. (Problem 10)
- Erasure requests can hurt fairness. On census data, skewed requests drop the model's true-positive rate for women while accuracy does not move. (Problem 12)
- Refusal is not forgetting. A language model trained to say "(withheld)" for Dana's phone number can still be made to give it up, by prefix forcing or a little relearning. (Problem 16)
- Even an exact deletion can leak. Comparing a recommender's public pages before and after a deletion tells an attacker what the deleted user watched, unless fresh noise is added. (Problem 18)
- Newer methods are not automatically better, and a privacy test can be fooled. Fourteen methods sit one fixed exam: no approximate method forgets Carol, and methods that make the model less sure of everything look "forgotten" until the test's reference models are built the same way. (Problem 19)
- Retraining for every request does not scale. The yearly bill grows with the square of the service size. SISA with 20 shards serves a year of 104 requests for about 2% of that cost and still matches, bit for bit, a fresh SISA model trained with the same shards and recipe, while exact Newton updates need a Hessian that a deep model cannot fit in memory. (Problem 20)
- Deleting the document is not deleting the person. In a RAG chatbot, deleting Fiona's tickets left her data in five of six stores, her words could be read back from her stored vectors and "deleted" index rows stayed on disk until compaction. Deleting by ID everywhere, redacting a mention in another ticket and compacting gives a stack that is bit-for-bit the one built without her, but a model trained on the same tickets still recites her phone number. (Problem 21)
- A surgical edit changes what a model says, not what it knows. Removing one direction switched off a habit, as in the November 2026 weight-editing news. But switching off neurons, removing directions and a ROME rewrite never took one person's phone number out. A new last layer, trained without her data, read it back from every edited model, and the same one-direction edit undid a refusal. (Problem 22)
You do not have to do all 24 notebooks. Find the row that sounds like you. Each route takes the problems in order.
| You are... | Do these problems | Time |
|---|---|---|
| New to unlearning and want the whole idea | 00 → 01 → 04 → 06 → 07 → 08 → 11 |
~4 hrs |
| A privacy, legal or governance person who needs to judge an unlearning claim | 00 → 04 → 06 → 11 → 12 → 17 |
~4 hrs |
| An ML researcher comparing unlearning methods | 00 → 04 → 05 → 06 → 08 → 09 → 10 → 11 → 19 |
~6 hrs |
| An LLM engineer who needs a model to forget facts or a customer's fine-tune | 00 → 04 → 06 → 11 → 15 → 16 → 16b → 21 → 22 |
~7 hrs |
| A platform or MLOps engineer designing systems that can delete | 00 → 06 → 07 → 11 → 15 → 17 → 18 → 20 |
~6 hrs |
| Doing the full course | everything, in order | ~21-30 hrs |
Times use the estimate at the top of each notebook and add a little for the tasks. The ROADMAP has the full stage-by-stage plan and a checkpoint for each stage.
- Python and NumPy at a working level; basic scikit-learn.
- Intermediate machine learning: what a loss function, gradient descent and overfitting are.
- No deep-learning framework needed. The neural networks are written in pure NumPy, so you can read every gradient.
- No legal background needed. Each legal idea is explained where it is used.
python -m venv .mulvenv
source .mulvenv/bin/activate # Windows: .mulvenv\Scripts\activate
pip install -r requirements.txt
python tests/test_mul.py # gradient checks + exactness proofs, ~1 minuteNo install? Every notebook has an Open in Colab badge. Run its first cell and it fetches the course library.
Every notebook runs end to end on a laptop CPU, most in under a minute. The small datasets (UCI Adult, SMS Spam,
Olivetti faces, sklearn digits, MovieLens) download once into data/. Only the optional Problem 16b needs PyTorch
(pip install -r requirements-optional.txt).
The problems are one argument, climbed one rung at a time. Each notebook ends with an ❓ Open question that the next one answers. Lower rungs are models where forgetting is a subtraction. The top rungs are LLMs, where nobody can yet do it reliably.
Start with 00-ReadFIRST.ipynb 
Click a stage to see its problems.
🪜 Stage 1 · Classical ML: when forgetting is a subtraction · Problems 1-3 · ~1.5 hrs
-
Problem 1: k-NN and Naive Bayes. The model is the data, or a sum over it. Unlearning is a delete or a subtraction, and it is bit-for-bit exact.
-
Problem 2: Linear and Logistic Regression. Coupled sums. Ridge gets an exact rank-one downdate; logistic regression needs a Newton step (influence functions, certified removal), the first approximate method.
-
Problem 3: SVM and Random Forest. No gradient, but local influence. Non-support vectors cost nothing to forget; a forest only retrains the trees that saw the point.
🧠 Stage 2 · Neural networks: the problem · Problems 4-5 · ~1 hr
-
Problem 4: Deleting is not forgetting. Every example touches 94% of the weights, and unusual people like Carol are memorised (Feldman's long tail). The Problem 1-3 tricks all fail.
-
Problem 5: Membership inference. How an attacker tells whether your data was used. Loss attacks, and the per-example U-LiRA test that flags 95% of Carol's digits.
✅ Stage 3 · Exact unlearning for neural nets · Problems 6-7 · ~1 hr
-
Problem 6: Retraining and the gold band. Two retrains differ, so "forgotten" is a band, not a number.
-
Problem 7: SISA. Shard and checkpoint at training time; unlearning becomes bit-exact and, with a user-aware partition, costs ~5% of a retrain. Plus the recipe trap in auditing.
⚡ Stage 4 · Approximate unlearning: cheaper, and full of traps · Problems 8-10 · ~2 hrs
-
Problem 8: Fine-tune, gradient ascent, NegGrad+, and the traps. Only 1 of 46 configurations passes the gate.
-
Problem 9: SCRUB and SCRUB+R. A student that selectively disobeys its teacher. Works for class forgetting, but over-forgets or relapses on a memorised individual.
-
Problem 10: Bad Teacher. Fast (under 1% of a retrain), but random ≠ forgotten, and its own metric ranks it above the gold standard.
🔬 Stage 5 · Proof · Problem 11 · ~45 min
- Problem 11: The Unlearning Audit. One pre-registered protocol, five checks (FORGET, PRIVACY, UTILITY, COST, RELEARN), three request types, machine-readable evidence. The best-looking approximate method is suppressed, not removed.
🌍 Stage 6 · Real-world Responsible AI: fairness, faces, poison, fine-tunes, LLMs · Problems 12-16b · ~4 hrs
-
Problem 12: Right to be Forgotten on the Adult census data. Skewed erasure requests collapse the model's true-positive rate for women while accuracy does not move, and NegGrad+ adds a method-made bias. Fairness monitoring and mitigation.
-
Problem 13: Consent withdrawn in face recognition (Olivetti). Exact eigenface unlearning, neural identity forgetting, and why memory ≠ capability.
-
Problem 14: A poisoned SMS spam filter. A backdoor that exact unlearning cannot remove when only part of the poison is found, and a fix that works with 10% detection.
-
Problem 15: Fine-tuning and un-finetuning for an enterprise SaaS. Per-tenant LoRA adapters (deletion is exact and free), re-fine-tuning from checkpoints, task-vector negation, and the lineage cascade.
-
Problem 16: LLM unlearning. Forgetting a phone number a language model can recite. n-gram subtraction, gradient ascent, NPO, refusal training, exposure, prefix forcing and the relearn attack: refusal ≠ forgetting.
-
Problem 16b (optional, PyTorch): the same lesson on a small GPT-style transformer.
🏢 Stage 7 · Engineering: a deletion request, end to end · Problem 17 · ~45 min
- Problem 17: The enterprise playbook. Lineage registry, decision procedure, deadline planning, a re-verifiable unlearning certificate, owners for every threshold, and an Unlearning pillar for the RAI Scorecard.
🧩 Stage 8 · Extensions: the same ideas in new places · Problems 18-22 · ~4 hrs
-
Problem 18: Recommender systems on real MovieLens data. Erin, a user with unusual taste, is on other people's "also watched" pages. Item-kNN forgets her exactly with a subtraction; matrix factorisation needs a retrain; and comparing the public pages before and after a deletion reveals what was deleted, unless you add fresh noise.
-
Problem 19: The method zoo. Fourteen methods, from retraining to SalUn and SSD, sit Problem 11's exam on three requests, labelled by family (model-agnostic, model-intrinsic, data-driven). No approximate method forgets Carol, and the new methods' passes on the random request turn out to be a confidence trick that only a fairly built privacy test catches.
-
Problem 20: Unlearning at scale. A year of deletion requests, two a week. Retraining for every request costs 104 retrains; batching monthly costs 13; SISA with 20 shards costs about 2, and a formula predicts how many shards each batch touches. After the whole year SISA is still bit-for-bit equal to a fresh SISA model (same shards, same recipe) trained without the deleted users. Exact Newton updates do not drift, but their Hessian grows with the square of the weights.
-
Problem 21: RAG and vector stores. A support chatbot looks old tickets up instead of learning them. Deleting Fiona's tickets from the database leaves her phone number and hospital stay in the chunks, vectors, keyword index, answer cache and logs. Deleting by owner ID only writes tombstones until compaction, an approximate index keeps her in its centroids until a rebuild, and a mention in her neighbour's ticket escapes the tags. After deletion, redaction and compaction the stack is bit-for-bit the one built without her, while a model trained on the tickets still recites her number.
-
Problem 22: Surgical weight editing. In November 2026 an AI edited 0.016% of a model's weights and fixed a fault with no training. The same kinds of edit (switching off neurons, removing a direction, a closed-form ROME rewrite) are tried on Gita's phone number in a staff-directory assistant. Some change what the model says, none passes the gold band, and a new last layer trained without her data reads her number back from every one. The same scalpel undoes a refusal by changing 752 weights.
🏋️ Want to test yourself? PRACTICE.md has 45 unsolved problems in four tiers (Beginner, Amateur, Semi-Pro and Pro), each with a hint, references and a checkable "done when".
00-ReadFIRST.ipynb … 22-Surgical-Weight-Editing.ipynb the course (24 notebooks)
mul/ a small, readable "Machine UnLearning" library the notebooks share
nn.py NumPy MLP + Adam + LoRA adapters (hand-written, gradient-checked)
classical.py exact unlearning for k-NN, Naive Bayes, ridge, logistic (Newton), forests, SVM
methods.py retrain, SISA, fine-tune, gradient ascent, NegGrad+, SCRUB(+R), Bad Teacher,
random labels, SalUn, L1-sparse, UNSIR, SSD, last-layer re-solve
audit.py MIA, U-LiRA, gold band, relearn test, the gate
lm.py n-gram + neural character LMs, exposure, NPO
recsys.py item-kNN (exact unlearning), matrix factorisation, "also watched" pages, the diff attack
rag.py a RAG chatbot's stores (chunks, vectors, BM25, IVF, cache, logs), tombstones, compaction
edit.py surgical weight edits (switch off neurons, remove directions, ROME) and the layer-below attack
lab.py, data.py, viz.py, train.py, torch_models.py (optional)
tests/ gradient checks and exactness proofs: python tests/test_mul.py
assets/ diagrams (generated by assets/make_diagrams.py)
results/ audit evidence records, the certificate and the leaderboard (Problems 11, 17, 19)
resources/ 79 papers (links, licences, open-licence text), a catalogue, a citation audit,
and fetch_papers.py to download the PDFs
templates/ the starting notebook for a new problem (see CONTRIBUTING.md)
All papers below are catalogued in resources/, with a suggested reading order.
python resources/fetch_papers.py downloads the PDFs to the paths the notebooks cite.
📑 Foundational papers
- Towards Making Systems Forget with Machine Unlearning - Cao & Yang 2015 - Problem 1
- Certified Data Removal from Machine Learning Models - Guo et al. 2019 - Problems 2, 12
- Machine Unlearning (SISA) - Bourtoule et al. 2019 - Problem 7
- Does Learning Require Memorization? - Feldman 2020 - Problem 4
📄 Papers behind Problems 5-22
- Membership Inference Attacks From First Principles (LiRA) - Problem 5
- Inexact Unlearning Needs More Careful Evaluations (U-LiRA) - Problems 5, 11
- Towards Unbounded Machine Unlearning (SCRUB) - Problem 9
- Can Bad Teaching Induce Forgetting? (Bad Teacher) - Problem 10
- On the Necessity of Auditable Algorithmic Definitions for Machine Unlearning - Problems 11, 17
- Machine Unlearning Fails to Remove Data Poisoning Attacks - Problem 14
- LoRA and Editing Models with Task Arithmetic - Problem 15
- The Secret Sharer (exposure) - Problem 16
- Negative Preference Optimization - Problem 16
- Eight Methods to Evaluate Robust Unlearning in LLMs and Do Unlearning Methods Remove Information from LLM Weights? - Problem 16
- Recommendation Unlearning (RecEraser) and When Machine Unlearning Jeopardizes Privacy - Problem 18
- SalUn, Selective Synaptic Dampening, Model Sparsity Can Simplify Machine Unlearning, UNSIR and Amnesiac Machine Learning - Problem 19
- Machine Unlearning (SISA) and Certified Data Removal, read again for cost at scale - Problem 20
- Retrieval-Augmented Generation, Text Embeddings Reveal (Almost) As Much As Text and Privacy Issues in RAG - Problem 21
- Sparks of Weight-Space Programming (a blog post), Knowledge Neurons in Pretrained Transformers, Refusal in Language Models Is Mediated by a Single Direction and An Adversarial Perspective on Machine Unlearning for AI Safety - Problem 22
🧭 Papers behind the concept pages
- Understanding Black-box Predictions via Influence Functions, Influence Functions in Deep Learning Are Fragile, Studying LLM Generalization with Influence Functions and TRAK - Influence functions, Problem 2
- Locating and Editing Factual Associations in GPT (ROME), Mass-Editing Memory in a Transformer (MEMIT) and Can Sensitive Information Be Deleted From LLMs? - Model editing vs unlearning
- Descent-to-Delete and Remember What You Want to Forget - Certified unlearning
- Federated Unlearning, How to Efficiently Erase a Client in FL? - Federated unlearning
- Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs and Catastrophic Failure of LLM Unlearning via Quantization - When forgotten knowledge comes back
- Text Embeddings Reveal (Almost) As Much As Text - RAG, vector stores and caches
- How To Break Anonymity of the Netflix Prize Dataset - Recommender systems, Problem 18
- Language Models are Few-Shot Learners (GPT-3) - Why the law cares
📚 Books and surveys
- The Architect's Handbook: Engineering Responsible AI by Shreyan Basu Ray - §5.1 on machine unlearning and the RAI Scorecard (in
resources/book/). - A Survey of Machine Unlearning - Nguyen et al.
- Rethinking Machine Unlearning for Large Language Models - Liu et al.
⚖️ Regulation the course refers to
- GDPR Art. 17 (right to erasure), Art. 7(3) (withdrawal of consent), Art. 9 (biometric data), Art. 12(3) (one-month response, extendable by two more months)
- EDPB Opinion 28/2024 on AI models and personal data
- India DPDP Act 2023 s.12 · California CCPA §1798.105
- FTC orders requiring deletion of models: Everalbum (2021), WW/Kurbo (2022), Rite Aid (2023)
What is machine unlearning? Removing the influence of specific training data from an already-trained model, so it behaves like a model that was never trained on that data. Deleting the data from a database is not enough.
Why does Responsible AI need it? Privacy laws (GDPR Art. 17, India's DPDP Act, CCPA) give people the right to erasure, regulators such as the FTC have ordered companies to delete models built on unlawfully used data, and poisoned or copyrighted data has to be removed from deployed models. The law does not always demand that model weights change, and no regulator has approved any unlearning method as equal to deletion yet. See the cases at the top of this page.
What is the difference between exact and approximate unlearning? Exact unlearning (retraining, SISA, Naive Bayes subtraction, deleting a LoRA adapter) provably gives the same model as retraining without the data using the same setup. For SISA that means the same shards and training recipe, not an ordinary model trained on all the remaining data. Approximate unlearning (gradient ascent, NegGrad+, SCRUB, Bad Teacher, NPO) is cheaper but must be audited. See Problem 11.
How do you prove a model forgot? Compare it with models retrained without the data (the gold band), and attack it: membership inference (U-LiRA), the relearn attack, and for LLMs exposure and prefix forcing.
Can LLMs really unlearn? Often they only suppress: refusal training hides a fact that prefix forcing and relearning bring back (Problems 16 and 16b). Re-fine-tuning from a clean checkpoint is the reliable route.
Do I need a GPU or PyTorch? No. Everything runs on a CPU in pure NumPy; only the optional Problem 16b uses PyTorch.
Keywords: machine unlearning, right to be forgotten, data deletion, GDPR, responsible AI, AI privacy, membership inference, SISA, SCRUB, Bad Teacher, NPO, LLM unlearning, LoRA unlearning, un-finetuning, data poisoning, AI governance.
Found a bug, a wrong number or a confusing sentence? Want to add a new problem? You are very welcome. Read
CONTRIBUTING.md first. New problems start from
templates/problem-template.ipynb, so every notebook keeps the same five steps.
The course is free to read, run and learn from, and all rights in it are reserved. You may link to it, quote short passages with credit and teach from it by sending learners here. Copying or imitating the content, the notebooks or the website's design and code needs written permission. See LICENSE for the full terms. Versions published before 26 October 2026 were under the MIT licence. AI agents and assistants should read AGENTS.md.
If you use this course in teaching or research, please cite it. GitHub's "Cite this repository" button (from CITATION.cff) gives you BibTeX and APA.
─── ⋆⋅☆⋅⋆ ───