A practitioner's guide to how AI models are trained, evaluated, and deployed — written for PMs, ops leads, and technical generalists who work with AI teams but didn't come up through ML research.
By Nikhil Thomas A — Delivery PM & Fractional Head of Data
🔗 Portfolio · PM AI Hub · LinkedIn
You manage AI products, AI teams, or AI programmes. You understand what a model does, but you want a clearer picture of how it got there — and what the tradeoffs were.
This playbook is not a research paper. It's not a tutorial. It's a working reference — the mental models and vocabulary you need to have real conversations with ML engineers, evaluate vendor claims, design evaluation frameworks, and make resourcing decisions about AI projects.
| Chapter | Topic | What You'll Be Able to Do |
|---|---|---|
| 01 — Training Paradigms | SFT, RLHF, RLEF, DPO | Explain tradeoffs between training approaches; know which fits which problem |
| 02 — Evaluation Metrics | Pass@K, HumanEval, MMLU, HELM, BLEU, BERTScore | Choose the right eval for the task; spot when benchmarks are being gamed |
| 03 — Data & Annotation | Dataset design, IAA, data flywheels | Spec a labelling project; assess annotation quality |
| 04 — Model Monitoring | Drift detection, quality scoring, A/B evals | Know when a deployed model is degrading |
| 05 — Cost & Quality Tradeoffs | Fine-tune vs. prompt vs. RAG vs. scale | Make the right architectural call for the right problem |
Read it linearly if you're new to AI training concepts. Jump to a chapter if you're preparing for a specific conversation — a vendor evaluation, a team planning session, a board Q&A.
Each chapter follows the same structure:
- The one-line version — if you only have 10 seconds
- The mental model — the intuition behind the concept
- The details — how it actually works
- The PM lens — what this means for how you work
- Common misconceptions — the things people get wrong
- Key questions to ask — when working with an AI team or evaluating a vendor
Most AI documentation is written for ML engineers. The abstractions are backwards for everyone else — you get the maths before the intuition, the implementation before the motivation.
I wrote this after realising I was nodding along in AI planning sessions without actually understanding the tradeoffs being made. These are the notes I wish I'd had.
Found an error, an outdated reference, or a concept that could be explained better? Open an issue or a PR. This is a living document.
- data-portfolio — ML projects applied to delivery and ops problems
- pm-ai-hub — Practical AI workflows for Delivery PMs