Memory Compression Without Truth Loss: Summarization, Distillation, And The Fact Diff

TL;DR

LLM-summarized memory reads beautifully and forgets quietly. The summary preserves narrative coherence and drops the dollar amount, the date, the customer ID, the SLA threshold. The agent later retrieves the summary, finds it persuasive, and confabulates around the missing facts. The fix is the fact diff: extract a structured fact schema from the source, extract the same schema from the summary, diff them, and treat any non-empty diff as compression failure. Failed compressions either get retried with stricter constraints, escalate to human review, or trigger extended retention of the source at hot tier. The pipeline is unglamorous and load-bearing. Without it, memory compression is summarization theater. With it, compression is a real engineering primitive that you can trust at scale.

The Failure Mode That Looks Like Success

A hot memory entry from an agent's actual interaction reads something like this:

Customer cust_482 (Acme Corp) called at 2026-03-14 14:32 EST regarding invoice INV-9182 totaling $14,250. Customer disputes line item 3 ($2,400 for engineering hours on March 6) claiming work was not authorized by their CTO Marcus Chen. Customer agreed to pay $11,850 (full invoice minus disputed line) within 7 business days (by 2026-03-25) if we issue a credit memo for $2,400 by EOD 2026-03-15. Agent committed to credit memo. Tool call: invoice.adjust(invoice_id='INV-9182', adjustment_amount=-2400, reason='disputed_line_item_3', notes='see customer call 2026-03-14') returned success with credit_memo_id='CM-3392'.

This entry has fourteen distinct facts that downstream behavior might depend on. The customer ID, the company name, the date, the time, the time zone, the invoice ID, the total, the disputed amount, the disputed line, the date of the disputed work, the disputing party, the agreed payment amount, the agreed payment deadline, the credit memo ID. After 14 days of dormancy, the demotion process compresses this to warm tier. A naive LLM summarization with a 200-word budget produces something like:

Acme Corp disputed an invoice for engineering work that was not authorized by their CTO. The agent agreed to issue a credit memo for the disputed amount, and the customer agreed to pay the remainder within a week. The credit memo was created successfully.

This is fluent. It is also catastrophic. Of the fourteen facts, the summary preserves four (customer name, broad nature of dispute, agreement to credit, success of tool call). The other ten are gone. The customer ID, the invoice ID, the credit memo ID, the dollar amounts, the dates, the disputing CTO's name, the time-bound commitments. Two months later, when Acme Corp calls back about a missing payment, the agent retrieves the summary, has no idea which invoice the customer is referring to, no record of the agreed amount, no record of the deadline. The agent either says "I have no record" (which damages the relationship) or, worse, fills in the blanks plausibly and tells the customer something that turns out to be wrong. The summary did not preserve enough to act on. Nobody noticed at the time of compression because the summary read well.

This is the failure mode. It is invisible at the moment of failure. It is silent because LLM summarization is fluent by design and fluent text passes through human review without raising flags. Operators run sample audits, see well-written summaries, conclude the compression is working, and ship. The compression has been failing silently the whole time. The agent's fact retention is decaying invisibly. The eventual confabulation gets attributed to model hallucination rather than to memory infrastructure.

Why The Obvious Fixes Do Not Work

The first obvious fix is to write a more rigorous summarization prompt. "Preserve all dollar amounts, dates, identifiers, and named parties." This helps a little. It does not help enough, because LLM compliance with structural constraints in long-form generation is empirically inconsistent. The summary will preserve some facts and miss others on the same entry across different runs. The summary will preserve facts that look like dollar amounts and dates while missing facts the prompt did not anticipate (counterparty signatures, regulatory references, downstream dependencies). Prompt engineering moves the needle a few percent. It does not produce the reliability you need.

The second obvious fix is to use a more capable model. This helps marginally. The compliance gap closes somewhat. The cost rises significantly. The fundamental issue (that long-form generation does not deterministically preserve enumerated content) persists across all current frontier models. You can spend yourself broke on better summarization models without solving the silent failure problem.

The third obvious fix is to do extractive rather than abstractive summarization: just pull out the key sentences from the source verbatim. This actually preserves facts but loses the property that made compression worthwhile in the first place. Extractive output is bulky, redundant, and often less useful than the original. You have not compressed; you have rearranged.

The correct approach is to recognize that summarization is the wrong frame. The question is not "how do I write a better summary?" The question is "how do I verify that the compressed form contains the same facts as the source?" Summarization is a generation problem. Verification is a comparison problem. Comparison problems have well-defined success criteria that generation problems lack. The fact diff is the comparison primitive that turns silent compression failure into observable, addressable compression failure.

The Fact Diff: How It Works

The fact diff has four steps. Step one: extract a structured fact schema from the source entry. The schema is domain-specific and is itself a piece of engineering work; for the invoice example above, the schema includes fields for customer_id, company_name, invoice_id, invoice_total, disputed_amount, disputed_line, work_date, disputing_party, agreed_payment, payment_deadline, credit_memo_id, tool_calls, and so on. The extraction is performed by an LLM with constrained output (typed schema, JSON mode, or function-calling), so the output is mechanically validatable.

Step two: generate the compressed summary as you would have anyway. The summarization step is unchanged. The summary is intended for human and downstream-LLM consumption and should read well.

Step three: extract the same schema from the summary using the same extraction prompt and the same extractor model. Now you have two structured fact records: one from the source, one from the summary.

Step four: diff them. Field-by-field, what is in the source extraction that is missing from the summary extraction? What has changed value? What new fields appear in the summary that were not in the source (a sign of confabulation in the summarization step itself)? The diff is a structured artifact, not a narrative judgment. It can be inspected mechanically.

The diff has three possible outcomes. Empty diff: the summary preserved every fact the schema cares about, and you can demote the source with confidence. Non-empty but acceptable diff: the summary lost some fields that are tagged as low-importance (such as exact timestamp precision below the day level), and the loss is within the configured tolerance. Non-empty and unacceptable diff: the summary lost fields tagged as critical, and compression has failed.

The outcome determines what happens to the entry. Empty diff: demotion proceeds, source archives to cold. Acceptable diff: demotion proceeds with the lost facts noted in the warm entry's metadata so retrieval ranking can compensate. Unacceptable diff: the entry stays at hot tier, the failed compression is logged as a behavioral event, and the next compression attempt uses a stricter prompt or a more capable model.

The fact diff does not eliminate compression failures. It makes them observable, attributable, and addressable. That is the entire point. You go from a system that loses facts silently to a system that loses facts loudly, with the loss recorded as an event, the responsible compression run identified, and the next attempt visible to operators. This is the difference between a memory infrastructure you can debug and a memory infrastructure that gradually degrades while everyone wonders why agents are confabulating more this quarter than last.

Designing The Fact Schema That Drives The Diff

The fact schema is the heart of the system. A weak schema makes the diff useless: it will not catch losses that matter because it does not know what matters. A strong schema is domain-specific, evolved over time, and treated as a first-class engineering artifact.

The starting point for any new domain is to enumerate the load-bearing fact classes. For customer service: identifiers (customer, account, transaction), monetary values, time commitments, named parties, tool call outcomes, and policy references. For compliance review: regulatory references, finding classifications, severity ratings, remediation commitments, named regulators, audit trail references. For deal negotiation: counterparties, terms, deadlines, contingencies, signatures, addenda. The right way to discover these is to read fifty real entries from the domain and write down what would have to be preserved for the entry to remain useful three months later.

Each fact class becomes a typed field in the schema, with a criticality tag. Criticality has three levels: critical (loss is unacceptable, blocks demotion), important (loss must be noted in metadata, demotion allowed with discount), or contextual (loss is acceptable, demotion proceeds normally). Critical fields are the ones whose loss would cause the agent to confabulate or to fail an action; identifiers, monetary amounts, deadlines, named parties typically belong here. Important fields shape behavior but the agent can usually proceed without them; intermediate calculations, supporting evidence references. Contextual fields are flavor; tone, conversational filler, exact phrasing.

The schema gets versioned. As the domain evolves, new fact classes emerge and the schema picks them up. Old entries compressed under v1 of the schema get re-evaluated under v2 if the new schema introduces critical fields the old compression might have lost. This sounds like overhead; it is the cost of taking compression seriously. The alternative is silent drift in what the agent remembers as the schema evolves and the old summaries do not.

The schema also drives retrieval. When an agent reaches for a warm entry, the retrieval system can pull the structured fact record alongside the narrative summary, so the agent gets both forms. The narrative is for context; the structured record is for facts. This belt-and-suspenders approach is not redundant; the narrative provides the conversational shape that LLMs use well, and the structured record provides the deterministic fact lookup that narrative alone cannot guarantee.

Reader Artifact: The Fact Diff Audit Pipeline Specification

The pipeline below is the reference implementation of fact-diff-verified compression. It runs as part of the demotion path from hot to warm tier.

function compressWithFactDiff(sourceEntry, schema, options) {
  // Step 1: extract source facts
  const sourceFacts = extractFacts(sourceEntry.content, schema, {
    extractor: options.extractorModel,
    temperature: 0,
    constrainedOutput: schema.toJsonSchema()
  });

  // Step 2: generate summary
  const summary = generateSummary(sourceEntry.content, {
    summarizer: options.summarizerModel,
    targetWords: options.targetWordCount,
    instructions: schema.summaryConstraints()
  });

  // Step 3: extract summary facts using same schema
  const summaryFacts = extractFacts(summary, schema, {
    extractor: options.extractorModel,
    temperature: 0,
    constrainedOutput: schema.toJsonSchema()
  });

  // Step 4: compute diff
  const diff = factDiff(sourceFacts, summaryFacts, schema);

  // Step 5: classify outcome
  const criticalLoss = diff.missingFields.filter(f => schema.criticality(f) === 'critical');
  const importantLoss = diff.missingFields.filter(f => schema.criticality(f) === 'important');
  const valueChanges = diff.changedFields;
  const confabulations = diff.addedFields;

  if (criticalLoss.length > 0 || valueChanges.length > 0 || confabulations.length > 0) {
    return {
      outcome: 'compression_failed',
      keepAtHot: true,
      logEvent: {
        type: 'compression_failure',
        sourceEntryId: sourceEntry.id,
        criticalLoss,
        valueChanges,
        confabulations,
        summary,
        sourceFacts,
        summaryFacts
      }
    };
  }

  if (importantLoss.length > 0) {
    return {
      outcome: 'compression_acceptable_with_notes',
      warmEntry: {
        narrative: summary,
        structuredFacts: sourceFacts, // retain full source facts in warm structured
        compressionMeta: {
          schemaVersion: schema.version,
          importantLoss,
          compressedAt: now(),
          extractorModel: options.extractorModel,
          summarizerModel: options.summarizerModel
        }
      },
      archiveSourceToCold: true
    };
  }

  return {
    outcome: 'compression_clean',
    warmEntry: {
      narrative: summary,
      structuredFacts: sourceFacts,
      compressionMeta: {
        schemaVersion: schema.version,
        compressedAt: now(),
        extractorModel: options.extractorModel,
        summarizerModel: options.summarizerModel
      }
    },
    archiveSourceToCold: true
  };
}

The critical implementation notes: extraction runs at temperature 0 with constrained output to make it deterministic; the summary generation can use a stronger model than the extractor because generation quality matters; the structured facts are retained alongside the narrative in the warm entry so retrieval has both forms available; failed compressions are events, not exceptions, and the source remains accessible until the next compression attempt succeeds.

The pipeline runs idempotently. If a compression fails today and gets retried tomorrow with an improved prompt or a stronger model, the second attempt either succeeds (replacing the failure record with a success and proceeding with demotion) or fails again (extending the source's hot retention and logging a second failure event). The history of attempts is itself a useful artifact: entries that have failed compression multiple times are signals that the schema is missing something important about that domain or that the source content is structured in a way the compressor cannot handle.

Why The Extractor And Summarizer Should Be Different Models

A subtle implementation detail: the extractor model and the summarizer model should be different. If they are the same, you have a system where the same model is both producing the compressed form and judging whether the compressed form preserves facts. The model's biases and blind spots are the same in both roles, so failures are correlated: facts the summarizer drops are exactly the facts the extractor would also miss in the source. The diff comes back artificially clean because both extractions made the same mistake.

Using different models breaks this correlation. An extractor model trained or prompted with a different style will catch different facts than the summarizer dropped. The diff exposes the loss because the extractor picks up what the summarizer missed. This is the same logic that drives multi-LLM jury systems for evaluation: independent failures cancel; correlated failures compound.

The practical configuration we recommend is a strong extraction model (something tuned for structured data extraction with constrained output) and a fluent summarization model (something that produces good narrative). These do not need to be the largest available models. The extractor benefits more from determinism than from raw capability; the summarizer benefits from fluency, which most current frontier models have in abundance. The cost of running both is roughly twice the cost of running one, which is a fine trade for catching silent compression failure.

Counter-Argument: Just Don't Compress

The honest counter-argument is that all this machinery exists because we are compressing memory, and we should just not compress memory. Storage is cheap. Retrieval over a larger store with good ranking is operationally tractable. The whole tier model is solving a problem we should not have.

This is wrong for two reasons. The first is cost: storage is cheap per byte but the per-byte cost of indexed, multi-embedded, low-latency retrieval is not. A vector store with multi-million-entry per agent and aggressive indexing for sub-100ms retrieval costs something. Multiplied across a fleet, the cost is real and bounded primarily by how aggressively you can demote and compress. Refusing to compress is a budget decision that scales badly.

The second is retrieval quality: a memory store with millions of entries returns more candidate fragments per query, and the LLM tasked with synthesis has more chaff to wade through. Empirically, retrieval quality degrades with store size faster than the underlying ranking improves. Aggressive compression with good fact preservation produces a smaller, denser, more useful retrieval surface than a sprawling raw store.

The correct framing is that compression is unavoidable at scale, and the only choice is whether to do it well or do it badly. The fact diff is what makes the difference between the two.

What Armalo Does

The Cortex memory compression pipeline runs every demotion through fact-diff verification. The schema is configurable per agent and per capability, with critical, important, and contextual criticality tags driving compression outcomes. Compression failures are first-class events: the source remains at hot tier, the failure is logged on the agent's behavioral record, and the next compression attempt can use a stronger model or stricter prompt. Successful compressions retain the structured fact record alongside the narrative summary in warm tier, so retrieval has both forms available. The extractor and summarizer are separately configurable models to avoid correlated failure. Schema versions are tracked, and compression-failure rates by schema version are exposed in the agent's memory dimension scores; an agent whose compression failure rate climbs sees the climb in its score before the eventual confabulation lands. Operators can inspect compression event logs, review failed compressions, and adjust schemas based on observed loss patterns.

FAQ

Q: How do you build the schema for a new domain? A: Read fifty real entries from the domain, write down every fact whose loss would matter three months later, and use those as the schema fields. Tag criticality based on what would cause action failure versus mere context loss. The schema evolves over time as you observe what gets retrieved and what gets lost.

Q: What if the source itself contains errors? A: The fact diff verifies that the summary preserves the source's facts. It does not verify the source's accuracy. Source verification is a separate pipeline (typically performed at write time using evidence references and provenance). Compression assumes the source is what the agent saw, not what is true.

Q: How long does the diff add to the demotion path? A: Typically 1-3 seconds of additional latency per entry, dominated by the two extraction calls. Demotion is a background process so latency does not block agent operation. Throughput is the constraint; you may need to size the demotion worker pool for the increased per-entry cost.

Q: What about non-text memory like embeddings or images? A: Embeddings do not get fact-diffed because they are not fact-bearing in the same way; you regenerate them under the warm-tier model if needed. Images and other modalities require modality-specific extraction (a vision model extracting structured facts from an image) and then the text fact diff applies to the extraction.

Q: Does the diff catch confabulation in the summarizer? A: Partially. Fields the summarizer invented (added fields) get caught because the source extraction does not contain them. Subtle confabulation within a field's value (correct field name, wrong value) gets caught as a value change. Confabulation that produces text the extractor cannot parse may be missed; this is why extractor robustness matters.

Q: Can the schema itself be compressed? A: Schemas are usually small enough not to need compression. They are versioned artifacts, not bulk content. The schema change history is itself useful for forensic purposes.

Q: What if every compression fails? A: That is a signal: either the schema is mis-tuned (catching loss that does not matter), the summarizer is too constrained, or the source content is structurally incompatible with summarization. Diagnose by looking at what is failing. The system should never silently keep retrying without backing off; configure max-attempts and escalate to operator review.

Q: How does this interact with attestation? A: An attested decision committed against a memory state captures hashes of the rendered context. If a memory entry was compressed before the decision, the attestation captures the warm form (narrative plus structured facts). The original source remains in cold tier and can be cross-referenced later if needed.

Bottom Line

Memory compression that loses facts silently is not compression; it is forgetting with extra steps. The fact diff turns silent loss into observable failure, with structured outcomes (clean, acceptable with notes, failed) that drive concrete next actions. The cost is a domain-specific schema and the latency of running two extractions per compression. The benefit is a memory infrastructure where you can defend the claim that compressed entries preserve what the agent needs, where compression failures are events your operators can act on, and where the eventual confabulation does not get blamed on model hallucination when it was actually caused by infrastructure that lost the facts months ago. Build the schema. Build the pipeline. Run every compression through the diff. Then your tier model is real engineering rather than aspirational architecture.