I accidentally turned LLM memory into program analysis(pwning.systems)
300 points by matt_d 12 days ago | 83 comments
tl;dr: Frustrated by LLM agents losing track of established facts during long vulnerability investigations, the author built Lemmalog, a Datalog engine that maintains structured facts, rules, and derivations with provenance tracking—so when an observation is invalidated, dependent conclusions are automatically retracted. On LongMemEval and LoCoMo benchmarks, it's competitive with dedicated memory systems (0.463 and 0.533 F1) while using ~38x less context than full-history prompting, and notably tops the field on knowledge-update questions. The takeaway: much of "LLM memory" is really a database problem (incremental evaluation, retraction, temporal facts) that classical program analysis techniques already handle well.
HN Discussion:
  • LLMs should be terminals for translation while formal reasoning handles the middle
  • ~This is a rediscovery of classic AI/Cyc approaches that have known limitations
  • Personal experience confirms LLMs fail to invalidate/retract outdated facts
  • Interest in applying this approach to other domains like business rules or hardware debugging
  • Sharing alternative or related tools/techniques (decision logs, DeepClause, graph diagrams)