RuleFit

RuleFit combines the models we have built so far. First, an ensemble of trees generates candidate decision rules. Each path becomes a binary feature: one when every condition is true, zero otherwise. Then a sparse linear model is fit using both the original features and those rule features.

This gives the linear model access to interactions. Instead of engineering young applicant AND missed payment by hand, a tree can discover that combination and pass it into the regression.

ℹ

Prediction equation

A regression RuleFit model has the form:

$\hat{y}=\beta_0+\sum_j\beta_jx_j+\sum_k\alpha_kr_k(x)$

$r_k(x)$ is a rule indicator. If the rule applies, its coefficient $\alpha_k$ is added to the prediction.

RuleFit Algorithm

A decision rule is turned into a binary feature: it returns 1 when every condition holds, 0 otherwise. That single number is then used as an input alongside the original features.

All conditions of rule are met

Otherwise

Does this applicant trigger the rule?

IF debt_ratio > 40% AND missed_payments ≥ 1

Low debt, no misses

debt 28% · missed 0

rule silent

High debt, no misses

debt 55% · missed 0

rule silent

High debt + misses

debt 55% · missed 2

rule fires

Decision rules are binary features. A tree ensemble can discover thousands of such rules from the data. RuleFit then selects the most predictive ones and assigns each a linear coefficient.

Decision rules are binary features: they encode an interaction (debt AND missed payments) as a single 0/1 input the linear model can use directly.

Lasso regularization is doing important work again. A tree ensemble can generate thousands of candidate rules. The sparse linear stage removes most of them, leaving a smaller collection of original effects and interactions.

⚠

Overlapping rules blur the story

One applicant may trigger several similar rules: debt above 40%, debt above 45%, and debt above 40% with a missed payment. You can still add the contributions exactly, but the explanation becomes repetitive and sensitive to small threshold changes.

Checkpoint

In RuleFit, what value does a rule feature take when all of its conditions are satisfied?

Checkpoint

When would you choose RuleFit over a plain sparse linear model?

←PreviousDecision RulesTrees and Rules Next→Prototype-Based Neural Networks (ProtoPNet)Neural Networks