When Bayesian neural networks remember too much

A neural network can fail at continual learning in two opposite ways. It can forget the past too quickly, or it can remember it too well.

The first failure is the well-known one. Learning a new task overwrites the representation of the previous one, and performance on old data drops: catastrophic forgetting. The second is less intuitive. A Bayesian posterior that accumulates every observation becomes more and more certain about the past. On a stationary task this is exactly what we want, since more evidence should mean less uncertainty. When the data distribution shifts, however, the model can be so certain about yesterday that it barely learns anything today. This has been called catastrophic remembering [2]: nothing the model has seen ever expires, so the past keeps its full weight in every update.

This post looks at the mechanism through two closely related Bayesian continual-learning rules: FOO-VB [1], which accumulates evidence indefinitely, and MESU [2], which limits how much past evidence survives. They differ by a single term, and most of what follows is about what that term does. The last part carries the same reasoning over to binary weights, where it leads to BiMU [3].

Bayes has no expiry date

Everything starts from Bayes' rule:

$$ p(w\mid\mathcal D) \propto p(\mathcal D\mid w)\,p(w). $$

After learning some data, we obtain a posterior. For the next batch, this posterior becomes the prior:

$$ p(w\mid\mathcal D_1,\mathcal D_2) \propto p(\mathcal D_2\mid w)\, p(w\mid\mathcal D_1). $$

Repeating this, and assuming the batches are conditionally independent given the weights, every observation keeps contributing to the posterior, all with the same weight:

$$ p(w\mid\mathcal D_{1:T}) \propto p(w)\prod_{t=1}^{T}p(\mathcal D_t\mid w). $$

This is what makes Bayesian continual learning attractive: no replay buffer is needed, since yesterday's posterior carries yesterday's information forward. The catch is that Bayes has no way of knowing whether yesterday is still relevant. On a stationary problem, the accumulation is exactly right. On a drifting stream, every old observation keeps pulling the posterior toward a solution that no longer exists.

For neural networks the exact posterior is intractable, so FOO-VB and MESU use variational inference: instead of computing the posterior, we look for the closest distribution \(q(w)\) within a simple family, closest in the sense of the Kullback-Leibler (KL) divergence, the standard measure of how far one distribution is from another. The family here is a mean-field Gaussian, meaning that each weight gets its own independent Gaussian. The memory mechanism is unchanged: today's posterior is tomorrow's prior.

A Bayesian weight is also a learning rate

For one Gaussian weight, we write

$$ q(w)=\mathcal N(\mu,\sigma^2), $$

where \(\mu\) is the current estimate and \(\sigma\) measures its uncertainty. Throughout, \(C\) is the cost of the current batch (its negative log-likelihood), and \(\partial C/\partial\mu\), \(\partial C/\partial\sigma\) are shorthand for the derivatives of the expected cost \(\mathbb E_q[C]\). For continual learning, \(\sigma\) has a second role, and it is the one that matters here:

$$ \Delta\mu \propto -\sigma^2 \frac{\partial C}{\partial\mu}. $$

The mean follows a gradient step whose learning rate is the variance. An uncertain parameter moves freely, while a confident one barely moves at all. Once a parameter has converged, this is what we want. If the posterior becomes too narrow, however, the model has effectively turned down its own learning rate: confidence first protects old knowledge, then starts blocking new knowledge.

FOO-VB keeps tightening the posterior

FOO-VB needs the derivatives of the expected cost with respect to \(\mu\) and \(\sigma\). The reparameterisation trick makes them computable by writing the random weight as a deterministic function of its parameters and an independent noise,

$$ w=\mu+\sigma\epsilon, \qquad \epsilon\sim\mathcal N(0,1), $$

so that the gradient can pass through the sampling. Writing \(g=\partial C/\partial w\) for the gradient at the sampled weight, the two derivatives are estimated as

$$ \frac{\partial C}{\partial\mu}=\mathbb E[g], \qquad \frac{\partial C}{\partial\sigma}=\mathbb E[g\epsilon]. $$

The diagonal version of FOO-VB then updates

$$ \mu\leftarrow \mu-\sigma^2\frac{\partial C}{\partial\mu}, $$ $$ \sigma\leftarrow \sigma\left[ \sqrt{1+\left(\frac{\sigma}{2}\frac{\partial C}{\partial\sigma}\right)^2} - \frac{\sigma}{2}\frac{\partial C}{\partial\sigma} \right]. $$

The mean update is the one from the previous section, with \(\sigma^2\) as the learning rate. For small updates, \(\sqrt{1+x^2}\approx 1\), and the variance dynamics reduce to

$$ \Delta\sigma \approx -\frac{\sigma^2}{2} \frac{\partial C}{\partial\sigma}. $$

To see what this does, consider the deliberately boring quadratic loss

$$ C(w)=\frac{h}{2}(w-\theta^\star)^2. $$

Under \(q\), its expectation is

$$ \mathbb E_q[C]=\frac{h}{2}\left[(\mu-\theta^\star)^2+\sigma^2\right], \qquad\text{so}\qquad \frac{\partial C}{\partial\sigma}=h\sigma, $$

and the variance update becomes

$$ \Delta\sigma \approx -\frac{h}{2}\sigma^3. $$

Writing \(v=\sigma^2\), so that \(\Delta v\approx 2\sigma\,\Delta\sigma\), and treating the update index \(t\) as continuous, this becomes

$$ \frac{dv}{dt}\approx-hv^2, $$

which integrates to

$$ v(t)=\frac{v_0}{1+hv_0t}, \qquad \sigma^2(t)\sim\frac{1}{ht}\quad\text{for large } t. $$

The variance never reaches zero, it just keeps shrinking: after \(t\) updates, the posterior precision \(1/\sigma^2\) has grown by \(ht\), one \(h\) per batch, exactly as Bayes prescribes. Since the mean update is multiplied by \(\sigma^2\), each batch makes the model harder to move. On a stationary task, this looks like convergence. On a drifting target, the model is not able to keep up: it stays too heavily constrained by the past to learn anew.

MESU adds forgetting

MESU [2] keeps the same data-driven updates and adds an explicit pull toward the original prior:

$$ \Delta\mu = -\sigma^2\frac{\partial C}{\partial\mu} + \frac{\sigma^2}{N\sigma_p^2} (\mu_p-\mu), $$ $$ \Delta\sigma = -\frac{\sigma^2}{2}\frac{\partial C}{\partial\sigma} + \frac{\sigma}{2N\sigma_p^2} (\sigma_p^2-\sigma^2). $$

Here, \(\mu_p\) and \(\sigma_p\) are the parameters of the original prior, and \(N\) controls the effective memory length. Each update now has two parts. The first is the FOO-VB update: a gradient step on the current data, with the variance as learning rate, starting from what the posterior already knows. The second pulls both the mean and the variance back toward the prior, at a rate set by \(1/N\), and it is the route through which old information disappears.

In probabilistic form, this corresponds to a tempered posterior update, an idea that goes back to the forgetting literature in adaptive estimation [9]:

$$ q_t(w) \propto p(\mathcal D_t\mid w)\, q_{t-1}(w)^{\,1-\frac1N} p(w)^{\frac1N}. $$

At every step, the previous posterior is raised to the power \(1-1/N\) and the prior to the power \(1/N\), so a fraction \(1/N\) of the accumulated evidence is traded for the prior. There is still no replay buffer, only a controlled decay of statistical evidence. In this form, the stability-plasticity trade-off is represented tangibly by two distributions: stability by the previous posterior, plasticity by the prior and the gradient signal, with \(N\) setting how the exponent is split between the previous posterior and the prior.

The variance gets a floor

For the same quadratic loss, the MESU variance update becomes

$$ \Delta\sigma = -\frac{h}{2}\sigma^3 + \frac{\sigma}{2N\sigma_p^2} (\sigma_p^2-\sigma^2). $$

At equilibrium, \(\Delta\sigma=0\), which gives

$$ \sigma_\star^2 = \frac{1}{hN+\sigma_p^{-2}}, $$

or, in terms of precisions,

$$ \frac{1}{\sigma_\star^2} = \underbrace{hN}_{\text{recent evidence}} + \underbrace{\frac1{\sigma_p^2}}_{\text{prior}}. $$

This is the memory window in a single equation. Each batch contributes a precision \(h\), so only about \(N\) updates' worth of evidence can remain consolidated, and older information is progressively relaxed. As \(N\to\infty\), the forgetting term vanishes, the variance floor goes to zero, and MESU recovers the cumulative behaviour of FOO-VB [2].

Forgetting also introduces bias

There is a catch: once the target stops moving, MESU never quite reaches it. The mean update has two parts, a gradient step with the variance as learning rate, and a pull toward the prior. The closer the mean gets to the minimum, the closer \(\partial C/\partial\mu\) gets to zero, and the more the prior term holds strength over the whole rule. The mean settles where the two cancel,

$$ 0= -h(\mu_\star-\theta^\star) + \frac{1}{N\sigma_p^2} (\mu_p-\mu_\star), $$

(the common factor \(\sigma^2\) has been divided out), which gives, using \(hN\sigma_\star^2=1-\sigma_\star^2/\sigma_p^2\),

$$ \mu_\star = \left(1-\frac{\sigma_\star^2}{\sigma_p^2}\right)\theta^\star + \frac{\sigma_\star^2}{\sigma_p^2}\mu_p. $$

A fraction \(\sigma_\star^2/\sigma_p^2 = 1/(1+hN\sigma_p^2)\) of the pull toward the prior always remains. Even at the optimum, the rule keeps the mean slightly away from it, in order to preserve plasticity. It is a form of variance regularization: the network is permanently unlearning a little and relearning a little, which is how it stays ready for an optimum that moves. This is the price of forgetting. A shorter memory gives faster adaptation, but more bias on stationary data. A longer memory preserves more information, but slowly brings back the rigidity.

Why curvature matters

In a neural network, parameters do not all see the same curvature. The second derivative of the cost along a parameter measures how strongly the data constrains it. A steep direction is strongly constrained, so its posterior variance shrinks. A flat direction remains uncertain, and therefore plastic. Once the prior term becomes negligible, the MESU equilibrium is approximately

$$ \sigma_\star^2 \approx \frac{1}{NH_D}, $$

where \(H_D\) is the diagonal of the Hessian, the matrix of second derivatives of the cost. This connects MESU to the intuition behind EWC [7] and Synaptic Intelligence [8], two regularization methods that protect important parameters from changing too much. The difference is that MESU also lets confidence decay: a parameter can become uncertain again once the old evidence is no longer supposed to dominate.

Binary weights: when the cloud becomes a coin

The same story appears in Bayesian binary neural networks, with a different geometry. A binary weight takes values

$$ \omega\in\{-1,+1\}, $$

with a Bernoulli posterior parameterised by \(\lambda\) [4]:

$$ q(\omega) = \frac{e^{\lambda\omega}} {2\cosh\lambda}. $$

Its mean and variance are

$$ m=\tanh\lambda, $$ $$ \operatorname{Var}[\omega] = 1-\tanh^2\lambda = \frac{1}{\cosh^2\lambda}. $$

At \(\lambda=0\), both signs are equally likely. As \(|\lambda|\) grows, one sign dominates and the posterior becomes nearly deterministic. \(|\lambda|\) plays the role of \(1/\sigma\) in the Gaussian case, on a logarithmic scale, since the standard deviation is about \(2e^{-|\lambda|}\), and in a much more constrained space: once the Bernoulli posterior saturates, the weight is both highly certain and difficult to flip.

For an ordinary gradient update, the chain rule through \(m=\tanh\lambda\) gives

$$ \frac{\partial\mathcal L}{\partial\lambda} = \frac{1}{\cosh^2\lambda} \frac{\partial\mathcal L}{\partial m}. $$

The factor \(1/\cosh^2\lambda\) shrinks as the weight becomes confident, so the gradient fades as the posterior saturates.

The Bayesian Learning Rule [6] uses the natural gradient instead. A natural-gradient step measures its size by how much the distribution changes rather than how much the parameter changes, which amounts to dividing the gradient by the Fisher information, the curvature of the KL divergence between the distribution and its close neighbours. For the Bernoulli in \(\lambda\), the Fisher information is exactly the variance \(1/\cosh^2\lambda\), so the natural gradient cancels this factor and the update becomes \(\Delta\lambda\propto-\partial\mathcal L/\partial m\) [4]. Under consistent evidence, \(|\lambda|\) then grows linearly with the number of updates, and the variance \(1/\cosh^2\lambda\approx 4e^{-2|\lambda|}\) collapses exponentially. The weight is now hard to flip because it has travelled far, not because its steps got smaller.

Bayes on a binary weight: evidence just adds up

BiMU [3] carries the bounded-memory idea of MESU over to binary weights, and adds metaplasticity, meaning that how easily a weight changes depends on its own state. In this line of work the weights are called synapses, a term borrowed from neuroscience [5].

When deriving Bayes' rule for continual learning with binary weights, we come to the conclusion that nothing prevents the runaway growth of \(|\lambda|\). The derivation is short enough to do by hand, on a single weight taken in isolation. Before the batch \(\mathcal D_t\), the posterior is \(q_{t-1}(\omega)\propto e^{\lambda_{t-1}\omega}\). Bayes' rule multiplies it by the likelihood \(p(\mathcal D_t\mid\omega)\), and since \(\omega\) only takes two values, that likelihood is just two numbers, \(p(\mathcal D_t\mid +1)\) and \(p(\mathcal D_t\mid -1)\). Any two positive numbers can be written as \(C\,e^{\delta_t\omega}\), with

$$ \delta_t = \frac12\log\frac{p(\mathcal D_t\mid +1)}{p(\mathcal D_t\mid -1)}. $$

\(\delta_t\) is half the log-likelihood ratio of the batch, meaning it measures how much more the batch favours \(+1\) than \(-1\). It is positive when the data prefers \(+1\), negative when it prefers \(-1\), and zero when the batch does not care. A batch twice as likely under \(+1\) gives \(\delta_t=\tfrac12\log 2\approx 0.35\). The constant \(C\) disappears in the normalisation, so multiplying the two exponentials simply adds the exponents:

$$ \lambda_t=\lambda_{t-1}+\delta_t, \qquad\text{so}\qquad \lambda_T=\lambda_0+\sum_{t=1}^{T}\delta_t, $$

where \(\lambda_0\) parameterises the original prior. This is why \(\lambda\) is called the natural parameter of the Bernoulli: Bayes' rule acts on it by plain addition. It is also the problem. The prior of each batch is the posterior of the previous one, so the only prior pull on \(\lambda\) is toward the previous task, never back toward \(\lambda_0\). If the batches keep favouring the same sign by about \(\delta\), then \(\lambda_T\approx\lambda_0+T\delta\) grows linearly, which is the natural-gradient behaviour of the previous section.

The forgetting framework of MESU [2] changes one thing. Before the new evidence is added, the previous posterior is raised to the power \(1-1/N\) and multiplied by the prior raised to the power \(1/N\). On the exponents, this reads

$$ \lambda_t=\left(1-\tfrac1N\right)\lambda_{t-1}+\tfrac1N\,\lambda_0+\delta_t. $$

Two things happen here. The first part shrinks the accumulated evidence by a factor \(1-1/N\) at every batch, so evidence that is \(k\) batches old has been multiplied by \((1-1/N)^k\approx e^{-k/N}\), and after a few \(N\) batches it is mostly gone. The second part pulls \(\lambda\) back toward the prior. Under constant evidence \(\delta\), \(\lambda\) settles where the shrinking and the new evidence cancel,

$$ \lambda_\star=\lambda_0+N\delta, $$

which is \(N\) batches' worth of evidence: the memory window again.

For a whole network, this tempering is done in variational form. BiMU minimises an objective with three terms: the expected loss \(\mathcal L\) of the current batch, a KL divergence to the previous posterior weighted by \(1-1/N\), and a KL divergence to the prior weighted by \(1/N\) [3]. For any exponential family, the class of distributions the Bernoulli and the Gaussian belong to, the gradient of the KL divergence to the prior is the variance times the distance between natural parameters,

$$ \frac{\partial}{\partial\lambda}\,\mathrm{KL}\big(q_\lambda\,\|\,p\big) = \frac{\lambda-\lambda_0}{\cosh^2\lambda}, $$

so the forgetting term in the gradient is \((\lambda-\lambda_0)/(N\cosh^2\lambda)\). It is weighted by the variance, meaning that it regularizes the high-variance synapses and does very little to the ones that are already certain.

But that isn't enough, because the certain synapses are the problem. The fixed point \(\lambda_0+N\delta\) grows with the memory window. With \(N=1000\), batches that favour one sign by only \(\delta=0.01\) already put the fixed point at \(\lambda=10\), where the variance \(1/\cosh^2\lambda\) is about \(10^{-8}\).

Where the metaplasticity comes from

The update itself makes things worse. Setting the gradient of the objective to zero gives [3]

$$ \lambda_t-\lambda_{t-1} = -\cosh^2\lambda_t\,\frac{\partial\mathcal L}{\partial\lambda} -\frac{\lambda_{t-1}-\lambda_0}{N}. $$

Two things happen here as well. The second part is the relaxation toward the prior. The first part is the data term, with a learning rate of \(\cosh^2\lambda\) in front of it, which comes from dividing the gradient by the curvature of the KL term, the variance \(1/\cosh^2\lambda\). This learning rate basically asks for infinite updates when \(\lambda\) gets high.

In exact arithmetic, this is compensated. The gradient \(\partial\mathcal L/\partial\lambda=(\partial\mathcal L/\partial m)/\cosh^2\lambda\) vanishes at the same rate, and the step \(\cosh^2\lambda\,\partial\mathcal L/\partial\lambda=\partial\mathcal L/\partial m\) stays finite. Floating point does not do exact arithmetic. In fp16 (16-bit floating point), \(\tanh\lambda\) rounds to exactly 1 once \(\lambda\) passes about 4.6, the computed variance \(1-\tanh^2\lambda\) is exactly 0, and the prescribed learning rate is \(1/0\).

The same cancellation means that exact Bayes has no metaplasticity. For the same \(\partial\mathcal L/\partial m\), a weight at \(\lambda=8\) moves \(\lambda\) by as much as a weight at \(\lambda=0.1\): nothing consolidates, and nothing becomes harder to move. By the time the probability reaches 1, the learning rate is infinite and the uncertainty is gone, which is a strange combination for a learning rule to aim at.

The condition above is implicit, since \(\lambda_t\) appears on both sides. To turn it into an update, BiMU expands it to second order around \(\lambda_{t-1}\), meaning it keeps the terms up to the second derivative (Theorem 1 in [3]). Writing \(g=\partial\mathcal L/\partial\lambda\), the learning rate becomes

$$ \frac{1}{\eta} = \frac{1}{\cosh^2\lambda}+2g\tanh\lambda+\frac{\partial^2\mathcal L}{\partial\lambda^2}, $$

with all terms evaluated at \(\lambda_{t-1}\). The last term is the Hessian of the data term, and for binary weights it can be worked out on paper. Under a mean-field posterior, the expected loss is linear in each \(m=\tanh\lambda\), whatever the loss, because the expectation over a \(\pm1\) weight is a weighted sum of two values, with weights \((1\pm m)/2\). Writing \(\mathcal L=a+b\tanh\lambda\) along one weight gives \(g=b/\cosh^2\lambda\) and

$$ \frac{\partial^2\mathcal L}{\partial\lambda^2} = -2\tanh\lambda\;g, $$

which exactly cancels the \(2g\tanh\lambda\) term. The exact second-order update therefore has \(\eta=\cosh^2\lambda\) again: symmetric, without metaplasticity, and with a finite step for any loss. The problem is computational. The Hessian is too expensive to compute in a large network, and even when it cancels on paper, \(\eta=\cosh^2\lambda\) is \(1/0\) in fp16.

BiMU replaces the Hessian with a surrogate, \(1/\alpha_{\max}+2|g|\), chosen so that the learning rate stays between zero and \(\alpha_{\max}\) (Proposition 2 in [3]). The full update is

$$ \Delta\lambda = -\eta\left(g+\frac{\lambda-\lambda_0}{N\cosh^2\lambda}\right), \qquad \frac{1}{\eta} = \frac{1}{\cosh^2\lambda}+2|g|+2g\tanh\lambda+\frac{1}{\alpha_{\max}}. $$

Near saturation, the variance and \(g\) are both tiny, so \(\eta\approx\alpha_{\max}\). Since \(g=(\partial\mathcal L/\partial m)/\cosh^2\lambda\), the step becomes

$$ \Delta\lambda \approx -\frac{\alpha_{\max}}{\cosh^2\lambda} \left(\frac{\partial\mathcal L}{\partial m}+\frac{\lambda-\lambda_0}{N}\right). $$

Both terms are now multiplied by the variance, as both terms of the MESU mean update are multiplied by \(\sigma^2\). The factor \(1/\cosh^2\lambda\) that exact Bayes cancelled is back, and the more committed a weight is, the smaller its steps. Bounding the learning rate is therefore enough to produce metaplasticity: committed weights consolidate, and uncertain ones stay plastic.

The fixed point does not change, but the way \(\lambda\) approaches it does. Ignoring forgetting and under consistent evidence, \(d\lambda/dt\propto e^{-2\lambda}\), so \(|\lambda|\) grows like \(\tfrac12\log t\) instead of linearly, and the variance decays like \(1/t\) instead of exponentially. With \(\alpha_{\max}\,|\partial\mathcal L/\partial m|\) of order one, getting to \(\lambda=8\) takes about a million updates, where the exact update, with steps of the same size, needs eight. In practice, the weights keep some doubt. The update also stays finite in fp16, since the learning rate is capped at \(\alpha_{\max}\) even when the variance has rounded to zero.

Replacing the Hessian also removes the cancellation, so the \(2g\tanh\lambda\) term of the exact expansion now survives next to the \(2|g|\) of the surrogate, and the learning rate becomes asymmetric. When the gradient pushes \(\lambda\) against its current sign, the two terms add up to \(2|g|(1+|\tanh\lambda|)\) and the step is damped. When it pushes along the sign, they reduce to \(2|g|(1-|\tanh\lambda|)\), which vanishes at saturation. Updates that reinforce an established sign are therefore treated differently from updates that try to overturn it, as in [5], where updates that would flip the sign of a consolidated synapse are progressively attenuated.

How much this asymmetry matters depends on how large \(g\) is compared with \(1/\alpha_{\max}\). Since \(g\) itself carries a factor \(1/\cosh^2\lambda\), deep in saturation the variance factor does most of the protecting. Between the two, a consolidated weight does not flip because of one noisy observation; it takes persistent opposing evidence [3].

The two mechanisms play different roles. Forgetting sets where \(\lambda\) can end up, a fixed point at \(\lambda_0+N\delta\) rather than infinity, and the bounded learning rate sets how fast it gets there.

So how long should the memory be?

For now, \(N\) is a hyperparameter. Yet the stream itself contains information about how quickly the environment changes: a slowly drifting world should require less forgetting, and a rapidly changing one probably more. The difficult part is inferring this timescale from the data, since it depends on task complexity, on the similarity between regimes, on model capacity, on noise, and on the way the distribution actually shifts over time. Making the memory length adaptive is the natural next step.

What the toy experiment does not show

The animation only measures tracking. This is intentional, and it is also a limitation. A bounded-memory method should lose some performance on tasks that are genuinely obsolete, and real streams can change abruptly, change their curvature, revisit old regimes, or be very noisy. The toy example captures none of this.

The point is narrower: Bayesian learning can fail not only because it forgets too much, but also because it remembers forever. Sometimes, the posterior just has to be allowed to let go.

References

1. Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry, Task-Agnostic Continual Learning Using Online Variational Bayes With Fixed-Point Updates, Neural Computation 33.11 (2021): 3139–3177. arXiv:2010.00373.
2. Djohan Bonnet, Kellian Cottart, Tifenn Hirtzlin, Tarcisius Januel, Thomas Dalgaty, Elisa Vianello, and Damien Querlioz, Bayesian continual learning and forgetting in neural networks, Nature Communications 16, 9614 (2025). doi:10.1038/s41467-025-64601-w.
3. Kellian Cottart, Théo Ballet, Djohan Bonnet, and Damien Querlioz, Active Continual Learning with Metaplastic Binary Bayesian Neural Networks, Proceedings of the 43rd International Conference on Machine Learning, PMLR 306 (2026). arXiv:2605.30198.
4. Xiangming Meng, Roman Bachmann, and Mohammad Emtiyaz Khan, Training Binary Neural Networks using the Bayesian Learning Rule, Proceedings of the 37th International Conference on Machine Learning, PMLR 119 (2020): 6852–6861.
5. Axel Laborieux, Maxence Ernoult, Tifenn Hirtzlin, and Damien Querlioz, Synaptic Metaplasticity in Binarized Neural Networks, Nature Communications 12, 2549 (2021). doi:10.1038/s41467-021-22768-y.
6. Mohammad Emtiyaz Khan and Håvard Rue, The Bayesian Learning Rule, Journal of Machine Learning Research 24.281 (2023): 1–46.
7. James Kirkpatrick et al., Overcoming catastrophic forgetting in neural networks, Proceedings of the National Academy of Sciences 114.13 (2017): 3521–3526.
8. Friedemann Zenke, Ben Poole, and Surya Ganguli, Continual Learning Through Synaptic Intelligence, Proceedings of the 34th International Conference on Machine Learning, PMLR 70 (2017): 3987–3995.
9. Rudolf Kulhavý and Martin B. Zarrop, On a general concept of forgetting, International Journal of Control 58.4 (1993): 905–924.
10. Richard Kurle, Botond Cseke, Alexej Klushyn, Patrick van der Smagt, and Stephan Günnemann, Continual Learning with Bayesian Neural Networks for Non-Stationary Data, International Conference on Learning Representations (2020).