Learning Inductive Bias with Hierarchical Bayesian Models

0.The Core Problem

Before we write any equations, it helps to step back and ask three questions about any learning system — questions that will guide the rest of this tutorial.

What needs to be computed?

Intelligent systems — whether human children or machine learning models — constantly face situations where they must generalize from very little data. A child hears a new word once and can already extend it to novel objects. A doctor sees one patient with rare symptoms and draws on years of related cases to form a diagnosis. The computational goal is clear: use experience with related problems to learn new problems faster. This is sometimes called "learning to learn" or meta-learning.

The challenge is that "related" is doing a lot of work in that sentence. How related are two bat colonies? Two drug compounds? Two animal species? The system must somehow represent the structure of relatedness itself — not just the data from any single problem, but the pattern that connects problems to one another.

How is that knowledge represented?

Hierarchical Bayesian models answer this with layers of abstraction. At the bottom layer sits the raw data — observations from individual groups, categories, or experiments. Above that sit the parameters that describe each group. And above those sit hyperparameters — parameters of parameters — that capture shared structure across all groups. You can think of this as a ladder:

• Bottom rung — observed data: the specific measurements you've collected (marble colors, test results, gene expressions).
• Middle rung — group-level parameters: what's true about each individual group (this colony's infection rate, this category's shape template).
• Top rung — population-level knowledge: what's true in general across all groups (how much do colonies typically vary? which feature dimensions define categories?).

This multi-level representation is what gives hierarchical models their power. The top rung encodes abstract knowledge — knowledge that doesn't describe any single group but constrains what all groups can look like. When a new group appears with barely any data, this abstract knowledge serves as an intelligent starting point.

What mechanism makes it work?

The mechanism is Bayesian inference operating simultaneously at every level. Data from individual groups flows upward, refining the model's beliefs about population-level structure. That refined structure then flows downward, acting as an informed prior for each group — especially helpful for groups with sparse data. This bidirectional flow is what statisticians call partial pooling: each group's estimate becomes a weighted blend of its own evidence and the collective wisdom of the population.

With these three questions in mind — what is computed, how is it represented, and what mechanism does the work — let's see the framework in action, starting with the simplest motivating example.

1.Why Hierarchical Models?

The computational problem at the heart of this section is deceptively simple: how do you estimate a rate when you barely have any data? On its own, this is just an estimation problem. But when you have many related estimation problems — many groups, each with sparse data — the question transforms into something richer. Can information from one group help you with another? Under what conditions? And how much help is appropriate?

The problem with flat priors

Imagine a wildlife researcher studying disease prevalence across 12 bat colonies. She visits each colony and tests a small sample of bats. Some colonies yield only 3–5 tested animals. A flat (non-hierarchical) Bayesian model treats each colony independently:

With tiny samples the posterior for each colony is dominated by the prior — we learn almost nothing. And we completely ignore the fact that colonies in the same region might share similar infection rates.

The hierarchical alternative

A hierarchical model adds a layer of abstraction:

Now the $\theta_i$ are no longer independent. They are linked through the shared hyperparameters $\alpha$ and $\beta$, which encode population-level knowledge: "What do bat colonies typically look like?" Data from every colony informs this question, and the answer flows back to improve estimates for each individual colony — a phenomenon called partial pooling or shrinkage.

This chart is worth reading carefully — it illustrates the central mechanism of hierarchical models in a single picture. Four things are plotted for each of the 12 colonies:

• Dark diamonds — the true infection rates (which a real researcher wouldn't know, but we simulated them so we can evaluate each method).
• Red circles — the no-pooling estimates (raw MLEs). Each colony estimated in isolation. Notice how erratic they are, especially for colonies where the sample was small. Colony 11, for example, tested only 21 bats and got an MLE near 0.10 — far from its true rate of ~0.25.
• Blue dashed line — the complete-pooling estimate. Every colony gets the same number (~0.36). This ignores the reality that colonies differ.
• Green squares — the partial-pooling (hierarchical) estimates. Notice the faint green arrows: each green square is pulled away from the red circle and toward the blue line. This pull is shrinkage.

The magic is that shrinkage is adaptive. Colonies with large samples (and therefore reliable MLEs) barely move. Colonies with tiny samples — where the MLE is essentially noise — are pulled strongly toward the population mean. The model is saying: "I don't trust your small-sample estimate much, so I'll lean on what I've learned from all the other colonies."

All code in this tutorial is available as a runnable Jupyter notebook.

2.The Beta-Binomial Model: Learning to Learn

The shrinkage chart showed us that hierarchical models help. Now we need to understand how they help — what representation makes this possible and what computation is actually being performed.

The answer lies in a simple but powerful idea: the model doesn't just estimate individual group parameters. It simultaneously learns what kind of groups are likely to exist in the first place. This "knowledge about knowledge" — captured in the hyperparameters — is what transforms a collection of independent estimation problems into a single coherent learning system.

We formalize this with the four-level hierarchy from Kemp, Perfors, & Tenenbaum (2007) [1], who developed this framework as a computational account of how human learners acquire abstract knowledge — what they called overhypotheses — from experience with concrete categories:

Consider two scenarios of "bags of marbles":

• Scenario A (homogeneous): 20 bags — 10 nearly all-black, 10 nearly all-white.
• Scenario B (heterogeneous): 20 bags — each with 10–40% black marbles.

Read this chart as a "before and after" of the model's beliefs. The dotted gray line is the flat Beta(1,1) prior — complete ignorance, assigning equal probability to every possible proportion. The colored curves are the learned priors after the model has processed 20 bags of data.

On the left (Scenario A), the learned prior has a dramatic U-shape — probability mass is piled up at the extremes (near 0 and near 1). The model has absorbed the pattern that bags tend to be nearly pure. On the right (Scenario B), the learned prior is a narrow peak around 0.25. The model has learned that bags are mixed with a moderate black-marble proportion.

This is the representational payoff: the hyperparameters $\alpha$ and $\beta$ don't just summarize the average bag. They capture the shape of variation across bags — how spread out or concentrated the population is. A U-shaped prior and a peaked prior encode fundamentally different expectations about the world, and those expectations will drive completely different predictions for the next bag we encounter.

Interpretation:

• Scenario A — After seeing 10 nearly-all-black and 10 nearly-all-white bags, the model learns a U-shaped prior. It expects the next bag to be almost entirely one color.
• Scenario B — After seeing 20 mixed bags, the model learns a peaked prior centered around 0.25. It tempers new evidence with the learned expectation that bags are mixed.

Predicting a new bag

Suppose we observe 1 black marble from 1 draw in a brand-new bag. What's the posterior on $\theta_{\text{new}}$?

This histogram overlay is the punchline of the entire section. All three models receive exactly the same evidence — one black marble from one draw — but they reach wildly different conclusions:

• Gray (flat prior): the posterior is a gentle Beta(2,1) — leaning toward black but still very uncertain. This is the "no experience" baseline.
• Red (Scenario A): the posterior spikes near $\theta = 1$. One marble was enough. Why? Because the learned U-shaped prior already expects bags to be nearly pure. A single black marble is strong evidence that this is an all-black bag.
• Blue (Scenario B): the posterior barely budges from the learned prior's peak around 0.25. One marble doesn't override the strong expectation that bags are mixed.

3.Category Learning & the Shape Bias

We now move from marbles to a more cognitively rich domain. The computational problem shifts: instead of estimating a single rate, the learner must figure out which features of the world are worth paying attention to.

Children learning new words face a challenging induction problem. When a parent points to an unfamiliar object and says "that's a blicket," the child must decide which dimension of the object defines the category: its shape? color? texture? size? The space of possible generalizations is enormous — the child could group objects by any combination of features. Yet by ~24 months, children reliably generalize new words based on shape rather than color or texture.

This shape bias is not innate. Developmental psychologists have shown that it emerges from experience: children who learn more nouns develop the bias earlier, and training studies can accelerate its onset. The computational question is: what kind of learning process could produce such a bias from nothing more than exposure to a few dozen object categories?

The answer, again, is hierarchical structure. We model this with a separate Beta-Binomial for each feature dimension, learning a dimension-specific scale parameter. The upper level of the hierarchy doesn't describe any single category — it captures how categories work in general: which dimensions are reliable grouping cues and which are noise.

Read this heatmap column by column. Each group of 5 columns is one category (C1 through C6), and each row is a feature dimension. The key observation: the Shape row is a solid block of color within each category — all exemplars share the same shape value. Color and Texture, by contrast, look like random noise within each category. This is the statistical regularity the hierarchical model must discover.

This violin plot shows the posterior distribution of $\mu$ (the learned "match rate") for each feature dimension. A high $\mu$ means the model believes that members of a category are very likely to share the same value on that dimension. Shape's $\mu$ is concentrated above 0.9 — near certainty that shape is shared. Color and Texture sit much lower, reflecting their within-category variability. The model has extracted, from just 6 categories of 5 objects each, the abstract principle: shape is what matters for category membership.

Now comes the payoff: a generalization test. The model sees a single exemplar of a new category ("blicket") and must choose which of three test objects belongs to the same category — one matching in shape, one in color, one in texture.

The bar chart shows the model's choice probabilities. Shape dominates. Not because anyone told the model that shape is important, but because the hierarchical structure allowed it to notice the pattern across previous categories and propagate that discovery downward as a prior for the new category.

This is a striking result when you consider what's happening at each level of analysis. The computational goal is to identify which feature dimension generalizes a category. The representation is a per-dimension $\mu$ that acts as a learned "attention weight." The mechanism is Bayesian inference propagating the consistency of shape across categories up to the hyperparameter level, which then constrains the new-category prediction. The shape bias wasn't hard-coded — it was learned from the statistical structure of the environment.

4.Hierarchical Causal Learning

The computational problem changes again. In the marbles example, groups shared a continuous population distribution. In category learning, the shared structure was which feature dimensions matter. Now we ask: what happens when the shared structure is categorical — when groups fall into distinct types, and the learner must figure out which type a new group belongs to?

Consider a pharmacologist studying whether different compounds in the same drug class cause a particular side effect. After characterizing several known compounds, she encounters a new one. Can experience with previous compounds accelerate learning about the new one?

We simulate two classes: Class A (benign, ~0% side-effect rate) and Class B (problematic, ~50% rate). The population-level knowledge here is not a continuous distribution but a finite mixture model with $K = 2$ components. We fix $K = 2$ by domain knowledge — the pharmacologist knows compounds are either benign or problematic, so the number of classes is not something the model needs to discover. The model specification is:

Here $\text{rate}_A$ is the expected side-effect rate for benign compounds (prior centered near zero), $\text{rate}_B$ for problematic compounds (prior centered near 0.5), and $w$ is the mixture weight governing class prevalence. The model must discover what each class looks like and classify a new compound — even from a single data point.

Inference proceeds by marginalizing over class assignments. For each compound, the likelihood is a weighted sum of the within-class likelihoods: $p(y_i) = (1 - w) \cdot p(y_i \mid \text{rate}_A) + w \cdot p(y_i \mid \text{rate}_B)$. This marginalization means we never need to commit to a hard class label during training — the model carries forward full uncertainty about which class each compound belongs to.

The left panel shows the model's posterior belief about whether the new compound belongs to Class B (the problematic class). Despite only one observation, the distribution is meaningfully shifted — the model is doing much better than chance. The right panel shows the inferred causal strength: the expected side-effect rate of the new compound. Notice that the peak is near 0.5, not 1.0. A naive model that only looked at the raw data (1 side effect in 1 trial) would estimate a rate of 100%. The hierarchical model knows better — it has learned the causal schema and uses it to place the new compound in context.

This example highlights a different kind of abstract knowledge: not a continuous population distribution (Section 2) or attention weights over features (Section 3), but a categorical schema — "there are two kinds of compounds, and they have different causal profiles." The representation is a mixture model at the population level. The mechanism is the same Bayesian inference, but now it's performing classification and estimation simultaneously: figuring out which class the new compound belongs to while also estimating its causal strength. One observation is enough because the population-level structure constrains the answer.

5.Property Induction over Taxonomic Structure

We now make a significant conceptual jump. In every example so far, groups were exchangeable — their labels could be permuted without changing the model. But many real domains come with structured relationships among groups: a phylogenetic tree, a spatial map, a social network. The question becomes how to encode that structure so the model generalizes along the right dimensions. Gaussian processes give us a principled answer: they define a prior over functions that respects a known similarity structure, letting the model "borrow strength" not uniformly, but proportionally to relatedness.

Our final example introduces a fundamentally different kind of structure. In the previous sections, groups were exchangeable — one bat colony was, a priori, as good as any other. But in many real domains, groups have known relationships to each other. Animal species sit on a family tree. Languages belong to language families. Brain regions connect through neural pathways. The computational problem now becomes: how do you use the structure of relationships among groups — not just the data within groups — to guide generalization?

If horses and cows share enzyme X, how likely is it that mice also have enzyme X? Your intuition probably says: not very likely, because mice are quite different from horses. But what about elephants? Somewhat more likely — they're all large mammals. And gorillas? Somewhere in between. These intuitions reflect an implicit taxonomic tree in your mind, and they follow a clear principle: properties are expected to be smooth on the tree — closely related species are more likely to share properties than distant ones.

The hierarchical Bayesian approach makes this precise by encoding the tree structure as a covariance function. We use a Gaussian process (GP) with an RBF (radial basis function) kernel on taxonomic distances. The intuition is straightforward:

If you're comfortable with multivariate normals, a GP is a natural extension: instead of specifying a fixed covariance matrix between a few variables, you specify a kernel function that generates the covariance between any pair of inputs — in our case, species.

• Closely related species (small tree distance) have strongly correlated property values.
• Correlation decays smoothly with taxonomic distance — there is no hard cutoff.
• The length-scale $\ell$ controls how quickly that decay happens: a small $\ell$ means only very close relatives share properties; a large $\ell$ lets influence spread across the tree.

The kernel function makes this precise:

The resulting covariance matrix visualizes these prior beliefs — each cell shows how strongly the model expects two species to share properties, based purely on their position in the taxonomy:

This heatmap encodes the model's prior beliefs about which animals are likely to share properties. Bright cells (high similarity) mean the model expects those two species to be alike; dark cells mean they're expected to differ. Notice the block structure: the 2×2 bright squares along the diagonal correspond to our four taxonomic groups (Primates, Ungulates, Large Mammals, Small Mammals). Within a group, similarity is high. Between adjacent groups, it's moderate. Between distant groups (e.g., Primates and Small Mammals), it's very low. This matrix is the model's prior about how properties spread across the animal kingdom.

Now comes the induction test: we observe that Horse and Cow both have a novel property (say, enzyme X). We condition on this observation and ask: how likely is it that each other animal also has the property?

The bar heights reflect a smooth gradient of generalization. Horse and Cow (observed) are at 100%. Elephant and Rhino — the next closest group on the tree — receive moderate probability. The primates and small mammals, sitting farther away on the taxonomy, get much less. This is exactly the pattern human participants produce in property induction experiments: generalization follows the contours of the taxonomic tree.

6.The Big Picture: Levels of Abstraction

We've now seen hierarchical Bayesian models applied to four different problems: estimating rates across bat colonies, learning marble-bag priors, discovering the shape bias in category learning, rapid causal classification, and generalizing biological properties across a taxonomy. Each domain was different, but the underlying architecture was the same. Let's make that unity explicit.

Recall the three questions we posed at the start: what needs to be computed, how is it represented, and what mechanism does the work? Across all our examples, the answers share a common structure:

What differs across domains is the form of the population-level representation: a Beta distribution for rates, per-dimension scale parameters for feature attention, a mixture model for causal schemas, or a covariance function for taxonomic structure. The chart below makes this comparison visual:

Each subplot shows the four levels of the hierarchy for one domain. The colored bars represent learned quantities, while gray represents fixed assumptions. The key parallel: in every case, the system has a "Level 3" — a layer of abstract knowledge that is neither directly observed nor directly assumed, but inferred from the pattern of data across groups. This is the layer that makes learning to learn possible.

Connection to Meta-Learning

The "learning to learn" framing naturally invites comparison with deep meta-learning methods, which pursue the same goal with different tools.

MAML (Model-Agnostic Meta-Learning; Finn, Abbeel, & Levine, 2017 [6]) learns a neural network initialization that can be fine-tuned to a new task in a few gradient steps. The learned initialization plays the same role as learned hyperparameters in an HBM: both encode abstract knowledge that biases future learning toward likely solutions. Grant et al. (2018) [7] made this connection formal, showing that MAML's inner loop approximates hierarchical Bayesian inference under certain conditions.

Prototypical Networks (Snell, Swersky, & Zemel, 2017 [8]) learn an embedding space where classification reduces to nearest-centroid lookup. The learned embedding dimensions are analogous to the learned dimension weights in Section 3: both determine which features matter for grouping novel items.

The key difference is in the trade-off between calibration and scalability. HBMs yield calibrated uncertainty via exact or MCMC inference — posterior credible intervals mean what they say. Deep meta-learning trades some of that calibration for the ability to scale to high-dimensional inputs (images, text) and millions of tasks. The shared principle underlying both paradigms: extract task-distribution-level inductive biases from experience, then deploy those biases to accelerate learning on new tasks.

When Does This Break?

Key Takeaways

1. Abstraction enables transfer. The upper levels of the hierarchy capture knowledge that is shared across related contexts. This shared knowledge is what makes rapid learning possible.
2. Priors are learned, not just assumed. In traditional Bayesian models, the prior is a modeling choice. In hierarchical models, the effective prior for each group is derived from data — it's the posterior over hyperparameters propagated down.
3. The mechanism is partial pooling. Each group's estimate is a weighted compromise between its own data and the population-level summary. Groups with less data lean more heavily on the population.
4. Structure matters. The choice of hierarchy matters enormously — tree-structured priors for taxonomy, dimension-specific scale parameters for feature learning, categorical mixtures for causal schemas. The right structure enables the right inductive biases.
5. Learning to learn is Bayesian inference at the meta-level. When we say a child has "learned to learn," we mean that abstract knowledge (level 3+) has been acquired that shapes the priors used for concrete learning (levels 1–2). Hierarchical Bayesian models formalize this idea precisely.

Summary of Model Specifications

Beta-Binomial (Section 2)

Category Learning (Section 3)

For each dimension $d \in \{\text{Shape, Color, Texture}\}$:

Property Induction (Section 5)

where $d(i,j)$ is the taxonomic tree distance and $\ell$ is a length-scale parameter.

References & Further Reading

1. Kemp, C., Perfors, A., & Tenenbaum, J. B. (2007). Learning overhypotheses with hierarchical Bayesian models. Developmental Science, 10(3), 307–321.
2. Tenenbaum, J. B., Kemp, C., Griffiths, T. L., & Goodman, N. D. (2011). How to grow a mind: Statistics, structure, and abstraction. Science, 331(6022), 1279–1285.
3. Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013). Bayesian Data Analysis (3rd ed.). Chapters 5 & 15.
4. McElreath, R. (2020). Statistical Rethinking: A Bayesian Course with Examples in R and Stan (2nd ed.). An accessible introduction to hierarchical modeling.
5. Rasmussen, C. E., & Williams, C. K. I. (2006). Gaussian Processes for Machine Learning. MIT Press. Background for Section 5.
6. Finn, C., Abbeel, P., & Levine, S. (2017). Model-agnostic meta-learning for fast adaptation of deep networks. ICML 2017.
7. Grant, E., Finn, C., Levine, S., Darrell, T., & Griffiths, T. L. (2018). Recasting gradient-based meta-learning as hierarchical Bayes. ICLR 2018.
8. Snell, J., Swersky, K., & Zemel, R. (2017). Prototypical networks for few-shot learning. NeurIPS 2017.