When Bayesian neural networks remember too much

A neural network can fail at continual learning in two opposite ways: it can forget the past too quickly, or it can remember it too well.

The first failure is well known. Learning a new task overwrites the representation of the previous one, and performance on old data drops. This is catastrophic forgetting. The second one is less intuitive. A Bayesian posterior that accumulates every observation becomes increasingly certain about the past. On a stationary task, this is exactly what we want, since more evidence should mean less uncertainty. When the data distribution shifts, however, the model can become so certain about yesterday that it barely learns anything today. This phenomenon is referred to as catastrophic remembering [2]. The problem is not a lack of memory, but a memory with no expiry date.

In this post, we look at the mechanism through two closely related Bayesian continual learning rules: FOO-VB [1], which accumulates evidence indefinitely, and MESU [2], which deliberately limits how much past evidence survives. They differ by a single term, and that term is the whole story.

Bayes has no expiry date

Everything starts from Bayes' rule:

$$ p(w\mid\mathcal D) \propto p(\mathcal D\mid w)\,p(w). $$

After learning some data, we obtain a posterior. For the next batch, this posterior becomes the prior:

$$ p(w\mid\mathcal D_1,\mathcal D_2) \propto p(\mathcal D_2\mid w)\, p(w\mid\mathcal D_1). $$

Repeating this indefinitely, and assuming the batches are conditionally independent given the weights, every observation keeps contributing to the posterior:

$$ p(w\mid\mathcal D_{1:T}) \propto p(w)\prod_{t=1}^{T}p(\mathcal D_t\mid w). $$

This is one of the attractive properties of Bayesian continual learning: no replay buffer is needed, since yesterday's posterior carries yesterday's information forward. Unfortunately, Bayes has no way of knowing whether yesterday is still relevant. On a stationary problem, accumulating evidence is a feature. On a drifting stream, it becomes the failure mode.

For neural networks, the exact posterior is intractable. FOO-VB and MESU instead work with an approximate distribution \(q(w)\), a mean-field Gaussian, optimised with variational inference. The memory mechanism stays the same: today's posterior is tomorrow's prior.

A Bayesian weight is also a learning rate

For one Gaussian weight, we write

$$ q(w)=\mathcal N(\mu,\sigma^2), $$

where \(\mu\) is the current estimate and \(\sigma\) measures its uncertainty. Throughout, \(C\) is the cost of the current batch (its negative log-likelihood), and \(\partial C/\partial\mu\), \(\partial C/\partial\sigma\) are shorthand for the derivatives of the expected cost \(\mathbb E_q[C]\). For continual learning, a second interpretation of \(\sigma\) matters more:

$$ \Delta\mu \propto -\sigma^2 \frac{\partial C}{\partial\mu}. $$

The variance scales the update. An uncertain parameter moves freely, while a confident one barely moves at all. This is what we want once a parameter has converged. However, if the posterior becomes too narrow, the model effectively turns down its own learning rate: confidence first protects old knowledge, then starts blocking new knowledge.

FOO-VB keeps tightening the posterior

Using the reparameterisation trick

$$ w=\mu+\sigma\epsilon, \qquad \epsilon\sim\mathcal N(0,1), $$

and writing \(g=\partial C/\partial w\) for the gradient at the sampled weight, the derivatives with respect to the mean and the standard deviation are estimated as

$$ \frac{\partial C}{\partial\mu}=\mathbb E[g], \qquad \frac{\partial C}{\partial\sigma}=\mathbb E[g\epsilon]. $$

The diagonal version of FOO-VB then updates

$$ \mu\leftarrow \mu-\sigma^2\frac{\partial C}{\partial\mu}, $$ $$ \sigma\leftarrow \sigma\left[ \sqrt{1+\left(\frac{\sigma}{2}\frac{\partial C}{\partial\sigma}\right)^2} - \frac{\sigma}{2}\frac{\partial C}{\partial\sigma} \right]. $$

For small updates, \(\sqrt{1+x^2}\approx 1\) and the variance dynamics reduce to

$$ \Delta\sigma \approx -\frac{\sigma^2}{2} \frac{\partial C}{\partial\sigma}. $$

To see what this does, consider the deliberately boring quadratic loss

$$ C(w)=\frac{h}{2}(w-\theta^\star)^2. $$

Under \(q\), its expectation is

$$ \mathbb E_q[C]=\frac{h}{2}\left[(\mu-\theta^\star)^2+\sigma^2\right], \qquad\text{so}\qquad \frac{\partial C}{\partial\sigma}=h\sigma, $$

we obtain

$$ \Delta\sigma \approx -\frac{h}{2}\sigma^3. $$

Writing \(v=\sigma^2\), so that \(\Delta v\approx 2\sigma\,\Delta\sigma\), and treating the update index \(t\) as continuous, this becomes

$$ \frac{dv}{dt}\approx-hv^2, $$

which gives

$$ v(t)=\frac{v_0}{1+hv_0t}, \qquad \sigma^2(t)\sim\frac{1}{ht}\quad\text{for large } t. $$

The variance never reaches exactly zero, it just keeps getting smaller: after \(t\) updates, the posterior precision \(1/\sigma^2\) has grown by \(ht\), one \(h\) per batch, exactly as Bayes prescribes. Since the mean update is multiplied by \(\sigma^2\), the model becomes progressively harder to move. On a stationary task, this looks like convergence. On a drifting task, this looks like a model that has forgotten how to change its mind.

MESU adds forgetting

MESU [2] keeps the same data-driven updates, but adds an explicit relaxation toward the original prior:

$$ \Delta\mu = -\sigma^2\frac{\partial C}{\partial\mu} + \frac{\sigma^2}{N\sigma_p^2} (\mu_p-\mu), $$ $$ \Delta\sigma = -\frac{\sigma^2}{2}\frac{\partial C}{\partial\sigma} + \frac{\sigma}{2N\sigma_p^2} (\sigma_p^2-\sigma^2). $$

Here, \(\mu_p\) and \(\sigma_p\) are the parameters of the original prior, and \(N\) controls the effective memory length. Each update now balances three components:

• Current data, pulling the posterior toward what is observed now;
• Past knowledge, carried by the posterior itself;
• The original prior, providing a route for old information to disappear.

In probabilistic form, this corresponds to a tempered posterior update, an idea that goes back to the forgetting literature in adaptive estimation [9]:

$$ q_t(w) \propto p(\mathcal D_t\mid w)\, q_{t-1}(w)^{\,1-\frac1N} p(w)^{\frac1N}. $$

At every step, a fraction \(1/N\) of the accumulated posterior is pulled back toward the original prior. There is still no replay buffer, only a controlled decay of statistical evidence.

The variance gets a floor

For the same quadratic loss, the MESU variance update becomes

$$ \Delta\sigma = -\frac{h}{2}\sigma^3 + \frac{\sigma}{2N\sigma_p^2} (\sigma_p^2-\sigma^2). $$

At equilibrium, \(\Delta\sigma=0\), which gives

$$ \sigma_\star^2 = \frac{1}{hN+\sigma_p^{-2}}, $$

or equivalently

$$ \frac{1}{\sigma_\star^2} = \underbrace{hN}_{\text{recent evidence}} + \underbrace{\frac1{\sigma_p^2}}_{\text{prior}}. $$

This is the memory window in a single equation. Each batch contributes a precision \(h\), so only about \(N\) updates' worth of evidence can remain consolidated, and older information is progressively relaxed. As \(N\to\infty\), the forgetting term vanishes, the variance floor goes to zero, and MESU recovers the cumulative behaviour of FOO-VB [2].

Forgetting also introduces bias

The variance floor does not come for free. When the target stops moving, the MESU mean satisfies

$$ 0= -h(\mu_\star-\theta^\star) + \frac{1}{N\sigma_p^2} (\mu_p-\mu_\star), $$

(the common factor \(\sigma^2\) has been divided out) and therefore, using \(hN\sigma_\star^2=1-\sigma_\star^2/\sigma_p^2\),

$$ \mu_\star = \left(1-\frac{\sigma_\star^2}{\sigma_p^2}\right)\theta^\star + \frac{\sigma_\star^2}{\sigma_p^2}\mu_p. $$

The model never fully settles on the current target: a fraction \(\sigma_\star^2/\sigma_p^2 = 1/(1+hN\sigma_p^2)\) of the pull toward the prior remains. This is the price of forgetting. A shorter memory gives faster adaptation, but more bias on stationary data. A longer memory preserves more information, but slowly brings back rigidity. Here, the stability-plasticity trade-off is not an abstract slogan: it is literally written in the posterior variance.

Why curvature matters

In a neural network, parameters do not all see the same curvature. A steep direction is strongly constrained by the data, so its posterior variance shrinks. A flat direction remains uncertain, and therefore plastic. Once the prior term becomes negligible, the MESU equilibrium is approximately

$$ \sigma_\star^2 \approx \frac{1}{NH_D}, $$

where \(H_D\) is the diagonal Hessian of the cost. This connects MESU to the intuition behind EWC [7] and Synaptic Intelligence [8], where important parameters are protected from changing too much. The Bayesian twist is that MESU also lets confidence decay: a parameter can become uncertain again once the old evidence is no longer supposed to dominate.

Binary weights: when the cloud becomes a coin

The same story appears in Bayesian binary neural networks, with a different geometry. A binary weight takes values

$$ \omega\in\{-1,+1\}, $$

with a Bernoulli posterior parameterised by \(\lambda\) [4]:

$$ q(\omega) = \frac{e^{\lambda\omega}} {2\cosh\lambda}. $$

Its mean and variance are

$$ m=\tanh\lambda, $$ $$ \operatorname{Var}[\omega] = 1-\tanh^2\lambda = \frac{1}{\cosh^2\lambda}. $$

At \(\lambda=0\), both signs are equally likely. As \(|\lambda|\) grows, one sign dominates and the posterior becomes nearly deterministic. \(|\lambda|\) is then doing the job of \(1/\sigma\) in the Gaussian case, but in a much more constrained space: once the Bernoulli posterior saturates, the weight is both highly certain and difficult to flip.

For an ordinary gradient update, we have

$$ \frac{\partial\mathcal L}{\partial\lambda} = \frac{1}{\cosh^2\lambda} \frac{\partial\mathcal L}{\partial m}. $$

The factor \(1/\cosh^2\lambda\) shrinks as the weight becomes confident, so the gradient weakens as the posterior saturates.

The Bayesian Learning Rule [6] uses the natural gradient instead. The Fisher information of the Bernoulli in \(\lambda\) is exactly \(1/\cosh^2\lambda\), so the natural gradient cancels this factor and the update becomes \(\Delta\lambda\propto-\partial\mathcal L/\partial m\) [4]. Under consistent evidence, \(|\lambda|\) then grows linearly with the number of updates, and the variance \(1/\cosh^2\lambda\approx 4e^{-2|\lambda|}\) collapses exponentially. Either way, the problem remains: repeated evidence pushes the posterior toward a nearly deterministic sign, and from there, flipping it becomes difficult.

BiMU combines forgetting with metaplasticity

BiMU [3] transfers the bounded-memory idea to the Bernoulli setting. The update of \(\lambda\) contains a relaxation term of the form

$$ \Delta\lambda = \cdots -\frac{\lambda-\lambda_{\mathrm{prior}}} {N\cosh^2\lambda}. $$

Interestingly, \(1/\cosh^2\lambda\) is exactly the Bernoulli variance, so forgetting is directly tied to uncertainty, just as the MESU mean relaxation is scaled by \(\sigma^2\): uncertain weights drift back to the prior quickly, confident ones slowly.

BiMU then adds a second mechanism: a bounded, state-dependent learning rate. Updates that reinforce an established sign are treated differently from updates that try to overturn it. This is reminiscent of [5], where updates that aim to flip the sign of a consolidated synapse are progressively attenuated. A consolidated weight therefore does not flip because of one noisy observation, and persistent opposing evidence is required [3]. The two mechanisms play different roles:

• Forgetting prevents confidence from accumulating forever;
• Metaplasticity makes the speed of change depend on the current state of the synapse.

Together, they keep the binary posterior uncertain enough to remain adaptable, without letting every small fluctuation flip a learned sign.

So how long should the memory be?

For now, \(N\) is a hyperparameter. Yet the stream itself contains information about how quickly the environment changes: a slowly drifting world should require less forgetting, and a rapidly changing one probably more. The difficult part is inferring this timescale from the data, since it depends on task complexity, similarity between regimes, model capacity, noise, and the way the distribution actually shifts over time. Making the memory length adaptive is therefore a natural next step.

What the toy experiment does not show

The animation only measures tracking. This is intentional, but it is also a limitation. A bounded-memory method should lose some performance on tasks that are genuinely obsolete, and real streams can change abruptly, alter their curvature, revisit old regimes, or be very noisy. The toy example captures none of this.

The point is narrower: Bayesian learning can fail not only because it forgets too much, but also because it remembers forever. Sometimes, the posterior simply needs to be allowed to let go.

References

1. Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry, Task-Agnostic Continual Learning Using Online Variational Bayes With Fixed-Point Updates, Neural Computation 33.11 (2021): 3139–3177. arXiv:2010.00373.
2. Djohan Bonnet, Kellian Cottart, Tifenn Hirtzlin, Tarcisius Januel, Thomas Dalgaty, Elisa Vianello, and Damien Querlioz, Bayesian continual learning and forgetting in neural networks, Nature Communications 16, 9614 (2025). doi:10.1038/s41467-025-64601-w.
3. Kellian Cottart, Théo Ballet, Djohan Bonnet, and Damien Querlioz, Active Continual Learning with Metaplastic Binary Bayesian Neural Networks, Proceedings of the 43rd International Conference on Machine Learning, PMLR 306 (2026). arXiv:2605.30198.
4. Xiangming Meng, Roman Bachmann, and Mohammad Emtiyaz Khan, Training Binary Neural Networks using the Bayesian Learning Rule, Proceedings of the 37th International Conference on Machine Learning, PMLR 119 (2020): 6852–6861.
5. Axel Laborieux, Maxence Ernoult, Tifenn Hirtzlin, and Damien Querlioz, Synaptic Metaplasticity in Binarized Neural Networks, Nature Communications 12, 2549 (2021). doi:10.1038/s41467-021-22768-y.
6. Mohammad Emtiyaz Khan and Håvard Rue, The Bayesian Learning Rule, Journal of Machine Learning Research 24.281 (2023): 1–46.
7. James Kirkpatrick et al., Overcoming catastrophic forgetting in neural networks, Proceedings of the National Academy of Sciences 114.13 (2017): 3521–3526.
8. Friedemann Zenke, Ben Poole, and Surya Ganguli, Continual Learning Through Synaptic Intelligence, Proceedings of the 34th International Conference on Machine Learning, PMLR 70 (2017): 3987–3995.
9. Rudolf Kulhavý and Martin B. Zarrop, On a general concept of forgetting, International Journal of Control 58.4 (1993): 905–924.
10. Richard Kurle, Botond Cseke, Alexej Klushyn, Patrick van der Smagt, and Stephan Günnemann, Continual Learning with Bayesian Neural Networks for Non-Stationary Data, International Conference on Learning Representations (2020).