Decision Trees
Decision trees are my favorite machine learning algorithm because they model decisions in a familiar form: ask a question, follow an answer, ask the next question. In healthcare work, I could hand a small tree to a clinician and discuss the actual path a patient took through the model.
A tree has split nodes, branches, and leaves. The leaf contains the prediction. The explanation for one prediction is the sequence of conditions from the root to that leaf.
Walk One Applicant Through a Tree
Change the applicant, then step through the exact questions the model asks. The explanation is the path, including the branches the applicant never reaches.
Income ≥ $55k?
yes
Late payments ≤ 1?
...
Final decision
...
Tree Anatomy
Click a term to highlight it on the diagram and read its definition. Work through all eight to build a complete vocabulary before moving on.
CART Formula Walkthrough
The prediction formula looks dense at first. Step through each piece — by the end you'll be able to read the whole thing in one glance.
Step 1 of 7 — The prediction
ŷ
ŷ (y-hat) is the model's predicted value for a single input instance x. For a regression tree this is a number; for a classification tree it is a class label.
Example prediction
For this house (sqft=1200, rooms=3), the model outputs:
ŷ = $310k
That is the model's estimate of this house's price.
The traditional CART algorithm grows a tree from the top down. At each node, it tries candidate feature thresholds and chooses the split that produces the largest reduction in impurity.
For classification, a common impurity measure is the Gini index:
A pure node contains one class and has Gini impurity zero. A mixed node has higher impurity. CART chooses the split with the best weighted improvement across its children. For regression trees, the equivalent criterion is mean squared error — each leaf stores the group mean, and the tree picks the split that minimizes the weighted MSE across both children.
How a Split Is Chosen
Drag the split line left and right. The tree picks the position that most reduces Gini impurity — watch the weighted impurity fall as the two groups become purer.
drag the red line to move the split
Gini reduction
0.500
strong split
Why trees work well in practice
Trees capture thresholds and feature interactions directly. A credit rule can say income matters first, then payment history matters only for applicants above that income threshold. The data does not need to follow a straight line, and feature scaling barely matters.
A readable algorithm can grow unreadable
A tree with four leaves can fit on an index card. A tree with four hundred leaves is a maze. Smaller trees are usually more interpretable, so depth limits and pruning are part of the model design.
Trees are also unstable. A small change in the data can change the root split, and that one choice changes everything below it. Random forests and boosted trees reduce this variance by combining many trees, but the ensemble loses the direct path-level transparency of one small tree.
Why is a random forest not inherently interpretable?
Think of a recent decision you made. Which parts fit naturally into yes-or-no splits, and which parts would make the tree explode in size?