MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs

Ke Wang^*, Yiming Qin^*, Nikolaos Dimitriadis, Alessandro Favero, Pascal Frossard

EPFL
NeurIPS 2025
^*Indicates Equal Contribution

Abstract

We present MEMOIR, a scalable framework for lifelong model editing that adds new knowledge through a residual memory—a dedicated parameter module—while preserving the pre-trained model’s core abilities. During editing, MEMOIR sparsifies input activations with sample-dependent masks so each edit touches a distinct subset of memory parameters, reducing interference. At inference, it retrieves the most relevant edits by matching the sparse activation patterns of new queries to those stored during editing, enabling minimal overwrite and informed retention across many sequential edits.

MEMOIR: Method Overview

MEMOIR is a lifelong model editing framework where new knowledge is leveraged in a dedicated residual memory module. Below, we provide an intuitive overview of MEMOIR's core principles, each illustrated with an accompanying visualization. Namely, minimal overwriting during training and informed retention during inference.

First, we use a sparse and sample-dependent masking named TopHash avoids overwriting between different edits, as illustrated in the figure above. Specifically, MEMOIR selects the top-k elements of the input activation vector as a robust "fingerprint." This generates a binary mask selecting which memory columns to update. Semantically similar prompts naturally lead to similar top-k activation patterns. This is followed by a hashing operation, which encourages the model to consider a broader set of features, reducing the risk of overfitting on dominant features and mitigating catastrophic forgetting by distributing updates across the trainable parameters of the memory module.

During inference, informed retention extracts the relevant knowledge conditional to the current prompt. Speficially, MEMOIR first identifies the type of prompt during inference—whether it is an edited prompt, a rephrased version of an edited prompt, or a prompt irrelevant to the edits, based on the distance of the current mask with the saved masks of previous edits (as shown in the figure above). It then activates only the relevant knowledge in the memory.

Results and Analysis

The table shows theQ&A task results on the ZsRE dataset for a range of edits (T). Higher is better for all metrics. MEMOIR consistently achieves the most balanced and robust performance across reliability, generalization, and locality. Unlike other methods, MEMOIR maintains strong results even as the number of edits increases.

The figure further shows the editing performance as the number of edits increases until 3000 on ZsRE: (a) LLaMA-3, (b) Mistral. MEMOIR maintains stable and superior metrics across all edit sizes, outperforming competing approaches throughout.

BibTeX

@article{wang2025memoir, title = {MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs}, author = {Wang, Ke and Qin, Yiming and Dimitriadis, Nikolaos and Favero, Alessandro and Frossard, Pascal}, journal = {arXiv preprint arXiv:2506.07899}, year = {2025}, note = {Equal contribution: first two authors} }