Kellian Cottart

Kellian Cottart

Doctoral student in Bayesian AI and Neuromorphic engineering at Université Paris-Saclay | Data Sciences at Ecole Polytechnique (IPP2023) | Engineer's degree at Ecole Centrale d'Electronique

Neuromorphic Symposium: Bayesian Continual Learning and Forgetting in Neural Networks (15 minutes)

2 minute read

Date:

When neural networks are deployed in the real world, data does not stay stationary. Environments evolve, tasks change, and distributions drift. Standard neural networks suffer from catastrophic forgetting: when learning something new, they overwrite what they learned before. But there is a second, often ignored challenge: catastrophic remembering, when networks become too rigid, stop adapting, and lose their ability to fit new data, leading to overconfident predictions.

Symposium 1 Symposium 2 Symposium 3

From a neuromorphic and biological perspective, both are problematic: forgetting too much breaks memory, forgetting too little kills plasticity. The question is not how to avoid forgetting, but how to forget in models where adaptability, reliability, and efficiency must coexist.

1. Synapses Are Inherently Uncertain

In biological systems, synapses may not be fixed parameters but stochastic processes characterized by uncertainty. Experimental and theoretical neuroscience shows that:

  • synapses encode a form of confidence,
  • plasticity is modulated by the certainty of synaptic states.

Recent work formalizes synaptic plasticity as Bayesian inference, where:

  • learning corresponds to a reduction in uncertainty,
  • stability emerges naturally as confidence increases.

2. MESU: Self-Regulated Plasticity from Uncertainty

Each synapse is defined by:

  • a mean, encoding its current belief,
  • an uncertainty, encoding confidence in that belief.

MESU (Metaplasticity from Synaptic Uncertainty) introduces a departure from standard Bayesian Neural Networks: each parameter is represented as a probability distribution that must be updated according to:

  • the expected loss gradient to fit the current data,
  • a prior regularization term leading to gradual forgetting according to a memory window controlling the amount of evidence retained,
  • a regularization term on the current state of the mean-field variational distribution, maintaining information over time.

Plasticity becomes uncertainty-modulated:

  • highly certain synapses update slowly and retain information,
  • uncertain synapses remain adaptive and plastic.

This framework leads to a unified learning-and-forgetting mechanism that does not rely on:

  • task boundaries,
  • replay buffers,
  • explicit parameter freezing.

Plasticity is self-regulated continuously, at the level of individual synapses.

3. Forgetting as a Computational Resource

MESU introduces a bounded memory window, which limits the amount of evidence retained by the model.

Information outside this window is progressively forgotten, rather than abruptly overwritten. This mechanism:

  • prevents unbounded accumulation of certainty,
  • avoids premature loss of plasticity,
  • preserves epistemic uncertainty where adaptation is required.

Crucially, this directly addresses catastrophic remembering where models become rigid and cease to adapt.

4. Implications for Neuromorphic Systems

MESU aligns naturally with neuromorphic design principles:

  • learning with MESU is different synapse-wise,
  • plasticity is governed by uncertainty rather than gradients,
  • no explicit task boundaries are required.

Uncertainty acts as a learning regularizer, making MESU particularly well suited for adaptive and lifelong neuromorphic learning systems.