RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversations
arXiv:2610.01780 · Paper published 2026-10-01 · Original paper ↗
RealCompanion introduces a longitudinal benchmark derived from 27,218 real-world human-companion messages across up to 120 days, enabling systematic evaluation of an AI companion's selective memory retrieval, profile fact extraction, trait inferencing, and restraint when history is unnecessary.
Existing conversational memory benchmarks rely on synthetic dialogues or crowdworker scripts where user personas are declared in advance and past context is artificially engineered to matter on almost every turn. Consequently, these benchmarks evaluate basic memorization rather than natural interaction. They fail to measure whether an AI system can discern when historical memory is actually needed or avoid over-inferring unstated user traits.
How it works
Sample and anonymize messages Input: 27,218 raw messages across 10 long-term user-companion relationships. Operation: Account identifiers are pseudonymized and message text is rewritten to protect privacy while preserving session rhythm, after which verbatim user turns are sampled into proportional, enriched, and abstention strata. Output: Anonymized conversation logs and verbatim user probe messages. Why needed: Real-world conversation traffic provides natural occurrence rates for past memory demand.
Resolve and verify references Input: A verbatim user probe message and its preceding conversation thread. Operation: Stage A parses what past entity or event the user points at and names its candidate store; Stage B tests candidate evidence turns against raw transcripts, dropping any unsupported references. Output: Verified historical reference citations and an updated context locus. Why needed: Prevents fabricated or ungrounded memories from polluting the evaluation ground truth.
Assign context rules Input: Verified reference list and recent thread window from Stage B. Operation: Nineteen deterministic rules map the verified context shape into one of four context tiers (basic, intermediate, hard, edge). Output: Assigned context tier, category, and recorded rule text. Why needed: Guarantees context tier assignment is a transparent, re-computable function of evidence shape.
Ground and validate responses Input: Verified context lists, recent thread window, probe, and tier assignment. Operation: Stage D writes a reference reply conditioned strictly on verified context; Stage E validates response support and category alignment, demoting items that fail. Output: An evaluation item with a grounded reference reply and an auditable 5-stage reasoning trace. Why needed: Ensures reference answers are strictly derivable from supplied evidence.
Example from the paper
A user states in March 'I'd be managing six people', in April 'I applied. I'm not sure I want it', and in June 'I got the job'. For the probe 'I got the job', Stage A identifies the referent, Stage B verifies the April evidence turn showing stated uncertainty, Stage C assigns the intermediate tier, and Stage D generates the grounded reply: 'You got the position even though you were not sure about it. How is it?'. Unlike AI systems that assume the user was ambitious (an unstated trait) or merely repeat stated facts, this framework evaluates whether a companion respects explicit conversational evidence while distinguishing stated facts from unverified inferences.
Evidence
Reported evidence and its scope
Reported result
Test conditions
What it supports
Human users require historical context outside the immediate thread in only 3.4% of real messages, with required memories sitting a median of 2,157 messages back.
Measured on the proportional stratum (1,169 unconditioned user probe turns across 10 real-world companion relationships).
Demonstrates a distributional difference between real user behavior and synthetic setups in this sample, showing that synthetic benchmarks with ~100% memory demand do not reflect natural conversation traffic.
Detecting whether a user message needs past memory achieves an AUROC of only 0.530, and labeling context as 'retrieved memories' instead of 'earlier turns' increases false memory misfires by 10 to 14 percentage points.
Evaluated using a language model detector and context ablation prompts (C1–C4) on proportional probes.
Shows that tested language models struggle to decide when to retrieve past history, and prompt framing artificially induces memory misapplication.
LLM agent systems achieve 0.69 to 0.70 F1 on persona reconstruction with high recall (0.86–0.89) but lower precision (0.56–0.59), while token consumption differs 30-fold.
Three agent systems evaluated over full multi-month histories across all 10 participants.
Indicates that tested agent systems consistently over-infer unstated user traits regardless of model scale or token budget.
Limits
Small subject sample: RealCompanion covers 10 participant relationships from one companion application over four months, so findings may not generalize across broader user demographics or alternative AI platforms.
Anonymization through model rewriting: Every message was rewritten by a language model to strip direct identifiers, which alters exact phrasing despite preserving semantic context.
Single-annotator context judgments: The evaluation of whether referents were visible in the recent ten-turn window was conducted by a single reader rather than an independent multi-annotator panel.
Reduced sample for memory-conditioned metrics: Under the strict evaluation reading where on-screen referents are removed, 5 of 10 participants contribute no memory-bearing probes, widening confidence intervals for memory-conditioned metrics.
Terms used here
Restraint
An AI system's ability to refrain from retrieving or referencing past conversational history when the current user message requires no historical context.
Proportional Stratum
A random sample of user probe messages drawn without filtering for memory demand, used to measure natural occurrence rates in conversation traffic.
Reasoning Trace
An auditable 5-stage record (Stages A–E) documenting referent resolution, evidence verification, rule classification, grounding, and validation for each item.
Misfire Rate
The proportion of user messages requiring no past context where an AI system inappropriately retrieves or references past memories.
Visual summarySource and authors
Paper authors: Arman Behnam, Sunglyoung Kim, Jiayi Yu, Eric Huang, Liangwei Yang
arXiv:2610.01780 ↗ · Episode released 2026-10-12 · Explanation updated 2026-10-13
AgentGarten couples physics engines to a shared neural renderer through untextured 3D geometry maps. This separation lets AI agents interact with photorealistic virtual worlds in real time while keeping physical rules and object coordinates explicitly controllable in code.
2026-10-08 · AI agents · Computer vision · Efficient AI
RIDE pushes a student model beyond its reinforcement-learning teacher by training student hidden representations to overshoot teacher representations along the layerwise residual vector between the teacher and its pre-learning base model, bypassing noisy output-token sampling.
2026-09-29 · Reinforcement learning · Language models
By measuring student learning progress through divergence from initialization, this paper fits joint power laws that predict a student language model's peak test accuracy and learning speed from model scale and teacher accuracy before distillation begins.
2026-09-26 · Language models · Reinforcement learning · Efficient AI
papers.luk.media