Prototype-Based Neural Networks (ProtoPNet)

When a birder identifies a difficult bird, they rarely say, "A hidden vector told me so." They point to parts: this crown looks like a red-bellied woodpecker; this wing pattern looks like one too.

ProtoPNet builds that style of case-based reasoning into an image classifier. A convolutional network creates a spatial representation, a prototype layer compares patches from the test image with learned training-image patches, and the final layer combines that evidence into class scores.

ProtoPNet patches
Prototypes are human understandable patches that come directly from the training data set [Source]

ProtoPNet Architecture & Training

ProtoPNet stacks three components in sequence. Click any stage to read how it works and why it enables interpretable decisions.

inputimageConv layersfeature extractionPrototype layerpatch similarityFully connectedclassificationclassscores

Convolutional layers

Standard CNN backbone that converts the input image into a spatial grid of feature vectors. Each position in the grid corresponds to a receptive field — a patch — in the original image.

f : image → Z (H′ × W′ × D feature map)

These layers are shared across all classes and trained end-to-end. Common backbones include VGG or ResNet truncated before the classification head.

The conv layers encode parts, the prototype layer compares parts, and the final layer combines evidence with auditable weights.

The prediction process is:

  1. Encode the image into a grid of latent patches.
  2. For each learned prototype, find the most similar patch in the test image.
  3. Convert distance into a similarity score.
  4. Weight the prototype evidence for each class and add it.

The explanation is tied to the architecture: this patch in the test image looks like that learned patch from a training image.

◆

Where this could matter

In medical imaging, a prototype model could point to a region of tissue and a similar learned case. A specialist can then judge whether the comparison uses a real pathology pattern or an irrelevant artifact.

ProtoPNet architecture diagram showing prototype layer comparing image patches to learned training-image patches
The reasoning process of ProtoPNet in deciding the species of bird shown in the top image. In the first column, we see boxes drawn around the parts of the image that look like the prototype. The next column shows our prototype and then the next is the training image where the prototype comes from. Finally, we show an activation map visualization. The final column in the table shows the points that indicate this image is of a red-bellied woodpecker, which is based on the similarity score between original image and prototype multiplied by the class connection. [Source]
⚠

A visible prototype can still be wrong

The model may learn background texture, a scanner mark, or a patch that is visually similar for the wrong reason. Prototype inspection makes that failure easier to find; it does not prevent the failure.

ProtoPNet was designed for image classification, and specialized interpretable neural networks often have this property: their explanation mechanism is built for one task. Training is also more complex than fine-tuning a standard classifier. In practice, compare the value of built-in evidence with compute, accuracy, engineering time, and the quality of a simpler model plus careful post-hoc analysis.

Checkpoint

What does a high prototype similarity score tell you?

Checkpoint

A bird classifier's strongest prototype is a patch of sky behind the bird. What would you do next?

←PreviousRuleFitTrees and RulesNext→Monotonic Neural Networks (MonoNet)Neural Networks