Dataset Bias

Dataset bias appears when the data used by an AI system does not adequately represent the people, contexts, or outcomes the system will face. Bias can enter through collection, labeling, measurement, missingness, historical decisions, or deployment feedback loops.

Transparent data practices help teams see bias before it becomes a product failure. The key is to inspect slices, not only aggregate metrics.

Where Bias Enters the ML Lifecycle

Bias doesn't enter at one point — it moves in a loop. Outputs from deployment feed back into future data collection, so the cycle can reinforce itself. Click a stage on the ring to see which bias types originate there.

Data Collection
Labels & Features
Training & Evaluation
Deployment
Feedback Loop
self-reinforcing loop

Data Collection

Historical Bias

Data reflects existing biases in the world.

Collected data reflects existing biases in the world. The original Word2Vec models, trained on text from a particular era, embedded gender associations that mirrored those of the source corpus. “Engineer” pulls male. “Nurse” pulls female. You're not seeing a flaw in the algorithm — you're seeing the algorithm faithfully reproducing what was in the data, which faithfully reflected the world at the time. The data was the problem, not the math.

Bias types explored

1 / 6

Removing a sensitive column only addresses one entry point. A system can be historically biased, unrepresentative, poorly measured, disparately learned, misused in deployment, and self-reinforcing — all at once, at different stages of the loop.

Bias analysis starts with representation: who is included, who is missing, and why? Then it examines labels: were outcomes measured consistently, or do they reflect uneven access, enforcement, reporting, or annotation quality?

A model can perform well overall while failing a subgroup. Report performance by meaningful slices, especially where error costs differ.

One way to evaluate fairness and bias is subgroup evaluation. In subgroup evaluation, the focus is on evaluating the consistency and fairness of explanations across different subgroups or protected attributes (e.g., gender, race, age). Statistical parity measures computed across different subgroups serve as metrics for evaluation.

Subgroup Evaluation: Statistical Parity Across Groups

This synthetic loan model scores twelve applicants split evenly across two protected-attribute subgroups. Move the approval threshold and compare the selection rate — a statistical parity measure — and the dominant explanation feature for each group.

Group A

Selection rate

67%

Approved

4 / 6

Dominant explanation among approvals

income

Group B

Selection rate

33%

Approved

2 / 6

Dominant explanation among approvals

zip code

Statistical parity gap

33 pts

Explanations diverge across groups

yes

At this threshold, approvals in Group A are driven mainly by income, while approvals in Group B are driven mainly by zip code. Even if selection rates look similar, the model may be reasoning differently — and less defensibly — about one subgroup.

Equal aggregate accuracy can still hide unequal selection rates or diverging explanations across subgroups. Subgroup evaluation surfaces both: whether the model treats groups consistently, and whether it relies on different signals to do so.
Checkpoint

Why can removing a sensitive column fail to remove dataset bias?

←PreviousIncident ResponseAccountabilityNext→Shortcut LearningFailure Analysis