The Junk Drawer Problem: Curating an AI Memory Store So Recall Helps
The Junk Drawer Problem: Curating an AI Memory Store So Recall Helps
Picture the store you actually have: 50,000 entries and climbing, every agent run depositing another lesson on top. A task comes in, the agent queries memory, and the top hit is a one-line hotfix someone logged last March for a vendor integration that’s since been rewritten twice. It ranks above the durable principle that would have solved the task cleanly, because the hotfix was recalled more recently and the principle was logged once, quietly, and never touched again. The agent reads the stale entry as still-true, acts on it, and ships a worse answer than it would have with no memory at all. The store didn’t fail by losing data. It failed by keeping all of it, forever, weighted equally. Volume became noise, and recall started polluting decisions instead of improving them.
That is the junk-drawer failure, and if you own a memory store you have already met it — you just may not have named it yet. This piece is for the engineer who has to keep a knowledge store decision-improving as it grows past the point any human will read it: what mechanics make a store forget, collapse, doubt, and rank on its own, so the next recall surfaces the load-bearing thing instead of the loudest thing.
A store that adds and never curates
Every team building agents now bolts on a memory; almost none run anything over it after the write. The unspoken assumption is that more remembered is more learned — that a store climbing toward six figures is a store getting smarter. It isn’t. Accumulation is not learning. A store that only ingests is a log with a search box, and a log is the wrong thing to recall a decision from, because everything in it sits at equal weight regardless of whether it’s still true, duplicates forty other entries, or has had the evidence turn against it.
The damage is quiet, which is what makes it dangerous. An entry that was load-bearing in March is dead weight by June, but nothing told the store to let go of it. A claim later proven wrong keeps getting retrieved as fact, because writing the contradiction somewhere else never demoted the original. The single insight someone phrased eight ways across eight runs comes back as eight hits, which the agent reads as overwhelming agreement when it’s really one belief echoing. None of these announce themselves — they just slowly make the average recall worse, and a worse recall is worse than no recall, because it arrives wearing the confidence of the whole store behind it.
The fix is to stop treating the store as an archive and start treating it as a curated set you govern. You already keep the authoritative history — the runs, transcripts, and outcome streams — and that record should stay append-only and complete; nobody acts off it directly. The store the agent recalls from is a different artifact with a different job, governed by one unforgiving rule.
The test that earns a store its keep
Most ungoverned stores fail that test on a large fraction of their contents, because they were never built to ask it. The four mechanics below enforce it after the fact — each a different reason an entry stops deserving recall, and a way to act on it without an irreversible delete. They are not a pipeline you run once, nor a ladder of escalating autonomy. They are four independent forces — decay, compression with subsumption, contradiction handling, and ranking — that you keep running continuously, because the store keeps degrading continuously.
Decay: forgetting what the fleet stopped recalling
The first mechanic answers staleness. An entry can be perfectly recorded, perfectly retrievable, and wrong now, because the world it described moved on. The lesson didn’t corrupt; it went stale, and a store that never forgets keeps serving it forever.
The fix is to make recall itself a vote. Every entry carries when it was first written, when it was last recalled, and how many times — so each retrieval re-stamps it as still load-bearing. A decay pass then archives entries the fleet has stopped recalling, on a deliberately conservative two-condition rule:
PRUNE an entry WHEN last_recalled older than max_idle_days (default 30)
AND generality < min_generality (default 1)Both conditions have to hold, and that conjunction is the whole safety. A broad principle is never pruned for being quiet — it stays true even when rarely needed, so low generality is required before idleness can retire anything; a narrow one-off is retired only once the fleet has demonstrably moved past it. Archived entries drop out of recall entirely — they read as absent, raw history again — but they are not deleted, and re-stating an entry with fresh evidence revives it. Forgetting here is reversible un-forgetting waiting to happen, which is why you can afford to be aggressive: wrongly archiving a still-useful entry costs one re-statement, while never forgetting costs a store that rots.
Compression and subsumption: many lessons, one principle
The second mechanic answers volume and duplication together. When forty runs all teach the fleet the same thing about, say, marketplace-compliance checks, an ungoverned store holds forty entries — and recall pulls back a wall of near-identical hits that the agent misreads as forty independent confirmations. It isn’t forty signals. It’s one belief, logged forty times, silently over-weighted by sheer repetition.
Compression collapses that. The store clusters near-duplicates by text similarity and proposes a single higher-generality principle that subsumes the cluster, linking the specifics beneath it. One auditable belief replaces forty scattered anecdotes, the store shrinks, and every future recall on that topic gets sharper because the principle outranks the noise it absorbed. Two mechanical details make this real rather than hand-wavy:
- A similarity threshold, not a vibe. Entries cluster when their token overlap (Jaccard) crosses ~0.6, and a cluster has to reach a minimum size (3+ same-topic entries) before it earns a principle — so the store generalizes from a pattern, not a coincidence.
- Subsumption: the broader claim wins on write. When a principle already exists for a topic and a new write arrives, the higher-generality statement is kept and the narrower one never demotes it. A specific tweak can’t overwrite a general principle; only a more-general principle can take its place. That single rule is what stops the store from regressing toward specificity as it grows.
This is the highest-leverage mechanic and the most dangerous one, which is why it is never auto-applied. Collapse the wrong cluster and you’ve laundered a bad assumption into a principle the entire fleet now trusts. So compression runs propose-only: the generalization is a suggested parent, the originals stay reachable, and a curator approves the batch or unmerges it in one action. A janitor that can irreversibly merge is a vandal one bad cluster away from poisoning everything downstream of it.
Contradiction handling: doubting a claim the evidence turned against
The third mechanic answers refutation. A principle can be earned, validated, ranked high — and then new evidence starts fighting it. The wrong response is to silently delete it; the also-wrong response common to ungoverned stores is to write the new evidence as a separate entry and keep recalling the old claim as if nothing happened.
Correct contradiction handling does neither. When evidence contradicts a principle, the store attaches the conflict to that principle — the contradicting claim, its evidence, and a timestamp ride alongside the original — and crucially it does not edit or delete the principle. Resolution (replace it, narrow its scope, retire it) is a deliberate, separate act of curation, never an automatic overwrite. What changes immediately is recall: retrieving the principle now returns the conflict with it, so an agent sees the doubt before it trusts the claim. A standing worklist of every entry carrying a live contradiction becomes the curator’s queue.
This beats auto-deletion because a single contradicting run is often noise, not a refutation — a store that deletes on first conflict thrashes, dropping good principles on flukes. Holding the principle in visible doubt lets the evidence accumulate before anyone acts, while ensuring no agent recalls the contested claim as settled in the meantime. The default posture for anything contested is doubt it on recall, because acting on a refuted claim costs far more than briefly distrusting a sound one.
Ranking: surfacing the load-bearing entry over the loud one
The fourth mechanic is what ties the other three to the agent’s experience. Decay decides what’s even in the active set; compression decides how many entries cover a topic; contradiction decides what’s trusted. Ranking decides what comes back first — and first is what the agent actually reads under a context budget. A clean store still fails if recall surfaces the stalest, narrowest entry at the top.
So every entry carries a single score, and recall sorts by it:
score = generality × 2 + min(proven, 10) + recencyBoost
recencyBoost = 3 if recalled within 7 days, 1 within 30, else 0The weighting is the argument. Generality dominates (×2) — a broad, transferable principle earns its keep regardless of how hot any single week is. The proven count adds confidence but is capped at 10, deliberately, so a lucky hot streak on a narrow tweak can never out-shout a general principle. The recency boost rewards what the fleet is actively recalling without ever letting recency alone win. The net effect: a general, validated, recently-used principle ranks above a narrow, stale, one-off tweak every single time — which is the precise inversion of the junk-drawer failure we opened on. The score is also pure and clock-injected, so the ranking is deterministic and testable rather than a black box that shifts under you.
Here are the numbers a healthy store moves, with curation running versus not:
- Near-duplicate entries in the active set18–30%3–6%Compression
- Stale entries retired per week~0steady weekly batchDecay
- Contested claims recalled without their conflictmost~0Contradiction
- Decision-quality delta (curated store vs. raw)unmeasured+6–15%The payoff
The first three rows are levers you can pull every week; the fourth is the outcome that justifies pulling them. Run an A/B — one cohort recalling from the curated store, one from the raw dump — and measure the win rate. If that delta is flat, your curation rules are wrong and you should stop and retune them. If it’s positive and growing, the store is compounding into something a competitor renting the same model can’t reproduce: the validated, de-duplicated, doubt-aware body of knowledge your fleet earned.
Turning hygiene on without breaking recall
The mechanics are clear; the risk is the rollout. Switching on aggressive curation against a live store that agents are recalling from in production is how you turn a degrading store into a broken one overnight. Stage it so recall keeps working at every step and nothing irreversible happens until the numbers earn it.
The order is deliberate. Start with ranking and contradiction-on-recall — they change what surfaces and what’s trusted without removing anything, so the worst case is a sort order you tune, never lost knowledge. Add reversible decay next: archiving is un-deletable forgetting, so a wrongly-archived entry is one re-statement from coming back, and you can run it aggressively from day one. Pilot compression only after ranking and decay have a track record, and keep it propose-only forever. Hard deletion never graduates — archived entries read as absent, which gets you the benefit of forgetting with none of the risk of destroying evidence you later need.
And because a memory store is high-volume and often holds sensitive operational knowledge, run the curation pass close to the store on your own hardware — clustering a million entries nightly against a metered API is a cost you don’t need to pay, and the sensitive lessons never leave your boundary to be de-duplicated.
What we covered
A bigger memory was never the goal — a store that recalls the right thing was. Build curation to forget toward an empty active set rather than hoard toward a full one, make every edit reversible, weight the recall toward general and proven over loud and recent, and the store stops being the junk drawer your agents reach into and starts being the earned advantage they recall because it makes the next decision better.
References: retrieval-augmented generation patterns · vector-store similarity clustering · recency/decay & generality-weighted ranking · NIST AI RMF (trustworthiness, provenance) · propose-only / reversible-edit curation governance.