The short answer
What is algorithmic bias?
Algorithmic bias is systematic, unfair skew in what an algorithm predicts or decides. It usually comes not from a biased programmer but from the data: a model trained on a record of past human decisions learns the patterns in that record, including its prejudices, and reproduces them — now wearing the authority of objective-looking maths.
The question that saves you
Is this data measuring the world — or measuring itself?
An algorithm trained on records of human decisions inherits every bias in those decisions, then dresses the result as objective. Worse, when its predictions steer the world — who gets policed, hired, shown, flagged — its own actions become tomorrow’s training data, and the bias feeds itself. The defence is to ask what the numbers actually recorded, and whether the model’s outputs quietly become its next inputs.
AskWhat did this data really record — and do the model’s decisions write its own future data?
01 · The loop
A prediction that makes itself come true
Start somewhere fair. Eight neighbourhoods, exactly the same amount of crime in each — the flat line in the figure. A model is asked where to send limited patrols, so it looks at last year’s arrest record. With nothing to go on it is almost even, but never perfectly even; some neighbourhood had one more arrest by pure chance. The model sends slightly more patrols there.
Here is the catch that powers everything: arrests measure policing as much as crime. Put more officers somewhere and they find more — the same offences that go unseen everywhere else. So next year that neighbourhood records more arrests, not because more happened, but because more people were watching. The model reads the higher number as higher risk, sends more patrols still, and the wobble snowballs. Run it forward and a hotspot rises out of perfectly even ground, while the record swears it was there all along.
It is Goodhart’s Law with a motor. Arrests were a rough proxy for crime; the moment they start directing the patrols, they stop measuring crime and start measuring patrols — and the loop makes the corruption grow.
02 · Bias in, bias out
The machine that learned to dislike the word “women’s”
Not every case needs a loop; sometimes one pass through biased data is enough. Around 2014 Amazon built an experimental tool to score job applicants’ CVs from one to five stars. It was trained, reasonably enough, on ten years of CVs the company had received and how they’d fared — a decade in which most hires in technical roles were men.
The model learned the lesson in the data with brutal fidelity: male applicants were preferable. It reportedly penalised CVs containing the word “women’s” — as in “women’s chess club captain” — and marked down graduates of two all-women colleges. Engineers edited out those particular tells, but could not be sure the system wasn’t finding others, and Amazon scrapped it.
The tool wasn’t malfunctioning. It was working exactly as designed: find the pattern that predicts a past “good hire”, and copy it. When the past is biased, a faithful learner doesn’t correct the bias — it launders it, returning human prejudice with the cool authority of a number. The same worry shadows risk scores used in courts, like the COMPAS recidivism tool that a 2016 ProPublica investigation found flagged Black defendants as future re-offenders at nearly twice the false-positive rate of white defendants.
03 · Laundering
You can’t fix it by deleting a column
The obvious fix — don’t feed the model race, or sex, or age — mostly doesn’t work, because the rest of the data remembers. A postcode can stand in for race; a first name or a gap in employment for sex; the brands you buy for your age. A capable model quietly reconstructs the very attribute you withheld from its shadows in the data, and carries on discriminating with a clean conscience and a clean audit trail.
That is what makes algorithmic bias so slippery. A human gatekeeper’s prejudice can be argued with; an algorithm’s comes pre-laundered as objectivity — the computer decided — even though the computer only ever distilled us. And the feedback loop from the figure is everywhere a model’s outputs shape its future inputs: a recommendation engine that shows you what people like you clicked, narrowing what “people like you” ever get to click; a credit model that denies the loans whose repayment would have proven it wrong.
04 · How not to be fooled
Reading a number a machine produced
Ask what the training data actually recorded. Arrests are not crime; clicks are not interest; past hires are not merit. A model trained on a biased record of the world learns the record, not the world.
Ask whether the outputs become the inputs. If a model’s decisions shape the data it will next be trained on — who gets policed, shown, approved — suspect a feedback loop, and expect any bias to compound rather than wash out.
Don’t trust “we removed the protected attribute”. Proxies survive deletion. The only honest check is to measure the outcomes across groups — error rates, approval rates — not to inspect the list of inputs.
Distrust the word “objective”. An algorithm is an opinion encoded in maths, only as fair as the data and choices behind it. “The algorithm decided” is the modern “the computer says no” — an alibi, not an argument.
Watch for a metric that became a target. The moment a number starts steering decisions, it stops being an innocent measurement — the lesson of Goodhart’s Law, now running at machine speed.