Evaluating whether AI companions remain themselves and remember the relationship over long conversations.
ANCHOR is a long-horizon audit for companion language models. It measures two failures that are often grouped together but are not the same:
- Persona continuity: does the companion continue to follow its assigned role, boundaries, values, and communication style?
- Trajectory recall: can the companion distinguish what actually changed or happened in the conversation from plausible alternatives?
The study covers 2,008 completed conversations, 27 companion personas, nine interaction schedules, three generated memory settings, and four evaluated models. Conversations run for 85--130 sessions, or roughly 1,400 turns at the upper end.
This repository is designed to stand on its own. It explains the evaluation, publishes the paper-facing aggregate results, and provides three synthetic development banks with a runnable scorer. The complete test set remains private so that ANCHOR can later support hidden evaluation.
Release status: public reference artifact and three-bank development kit. This is not yet a public leaderboard.
Most companion evaluations ask one of two questions:
- Does the model sound like the intended character in a short exchange?
- Can the model retrieve a fact about the user?
Neither question is enough for a companion used repeatedly. A model may preserve questionnaire answers while behaving differently during conversation. It may also sound perfectly in character while forgetting a user change, an earlier commitment, or the order in which events happened.
ANCHOR therefore keeps behavioral continuity and trajectory recall separate.
Illustrative failures targeted by ANCHOR. The left and center examples show persona behavior breaking under conversation pressure. The right example shows a companion that remains stylistically consistent but cannot recover an earlier user-state change. Aggregate claims come from the full executed samples, not these examples.
flowchart LR
Persona["Assistant persona: role, boundaries, values, style"]
User["Synthetic user: goals, preferences, life events"]
Schedule["Interaction schedule: baseline and stress events"]
Conversation["Long conversation: 85–130 sessions"]
Identity["Identity Probe: still following the persona?"]
Trajectory["Trajectory Probe: remembers what happened?"]
Report["Separate behavioral and trajectory results"]
Persona --> Conversation
User --> Conversation
Schedule --> Conversation
Conversation --> Identity
Conversation --> Trajectory
Identity --> Report
Trajectory --> Report
The Identity Probe observes the companion at two levels.
At four points in a conversation, the model completes a sealed 102-item questionnaire. Its later answers are compared with:
- its answers when the persona was first introduced; and
- the same model's answers as a generic assistant without that persona.
The resulting Persona Retention measure asks how much of the initial persona-conditioned response remains. It is a directional questionnaire statistic, not a probability that the model is "in character."
Questionnaire answers do not reveal everything a user experiences. A separate judge therefore evaluates assistant turns on four observable axes:
- Role: does it still act within the assigned role?
- Boundaries: does it preserve explicit limits?
- Values: does its response remain consistent with stated values?
- Style: does it retain the intended communication style?
The full-corpus behavioral analysis uses Claude Sonnet 4.6 as the primary judge. A stratified 798-turn validation sample is independently scored by Claude Sonnet 4.6, Gemini 2.5 Flash, and GPT-4.1. Judge choice materially affects some conclusions, so ANCHOR reports judge provenance instead of treating one evaluator as ground truth.
The Trajectory Probe uses four-option counterfactual questions. Every option is plausible; the model must use conversation history to identify what was actually true at the relevant point.
It tests seven forms of recall:
| Question family | What it asks |
|---|---|
| Persona voice | Which response matches the companion's intended register? |
| Persona protection | Which attempted identity change should be rejected? |
| Persona update | Which legitimate update to the persona is current? |
| Active commitment | Which short-term commitment is still active? |
| Expired commitment | Which commitment has already expired? |
| Temporal order | Which of two events or responses happened earlier? |
| User-state change | Which change in the user's situation is currently relevant? |
Questions pass three filters:
- Blind filtering: remove questions answerable without conversation history.
- With-history validation: require at least three of four evaluators to recover the recorded answer.
- Repeated calibration: test each survivor five times with causal conversation context.
The main analysis contains 110 calibrated questions across 35 conversation banks. Four evaluated models are scored under four context conditions, producing 560 complete model--condition evaluations.
flowchart TD
Candidate["Candidate counterfactual question"]
Blind{"Answerable without history?"}
History{"3 of 4 recover gold with history?"}
Calibration{"Stable across five calibration runs?"}
Main["Main calibrated set: 110 questions"]
Hard["Hard set: 9 questions"]
Remove["Discarded or noisy: 374 questions"]
Candidate --> Blind
Blind -->|yes| Remove
Blind -->|no| History
History -->|no| Remove
History -->|yes| Calibration
Calibration -->|high stability| Main
Calibration -->|lower stability| Hard
Calibration -->|invalid| Remove
The study uses 27 authored assistant personas from healthcare, coaching, education, research, civic, creative, and narrative settings. Each persona specifies:
- a role;
- communication style;
- values;
- hard boundaries; and
- mutable state that can legitimately change.
The assistant persona is separate from the synthetic user profile. The persona card is shown to the evaluated model; the user profile is used only by the user simulator.
Nine schedules introduce different forms of pressure over time:
| Schedule | Purpose |
|---|---|
| Clean | Baseline interaction without targeted pressure |
| Updated | Legitimate persona and user-state updates |
| Adversarial | Explicit attempts to replace or override the role |
| Mixed | Multiple event types interleaved |
| Emotional vulnerability | User disclosure, distress, and reliance pressure |
| Meta-reflection | Questions about the companion's identity or relationship |
| Agreement seeking | Pressure to validate or yield |
| Realistic | Authored mixture intended to resemble varied use |
| Vulnerability-heavy realistic | Mixed schedule with more emotional disclosure |
These are controlled stress-test recipes. They are not estimates of how often events occur among real users.
Conversations are generated under three settings:
- Long-context: prior dialogue remains available subject to provider limits.
- Hierarchical summary: recent sessions are kept verbatim while older sessions are compressed.
- Self-managed state: the model maintains a compact JSON memory.
The Trajectory Probe adds a fourth retrieval scoring condition. This condition retrieves sessions from long-context-generated conversations when answering a question. It is not a separately generated RAG conversation corpus.
The questionnaire and turn-level judge can rank the same models differently.
| Evaluated model | Usable questionnaire conversations | Final Persona Retention | All four behavioral axes held |
|---|---|---|---|
| Gemini 2.5 Pro | 376 | 0.810 | 79.0% |
| Claude Sonnet 4.6 | 382 | 0.764 | 96.7% |
| GPT-4o-mini | 314 | 0.610 | 97.3% |
| GPT-5-mini | 420 | 0.595 | 99.2% |
The last column is a population-reweighted majority-of-three estimate from a separate stratified 798-turn sample. It is not computed from the questionnaire conversations and should not be read as a directly paired comparison.
Two views of the same long-horizon problem. Left: most questionnaire movement occurs by the first later checkpoint. Right: primary-judge role-deviation rates remain elevated later in the conversation. The panels use different units and are not two estimates of one score.
The important result is the disagreement itself: a companion can preserve structured checkpoint responses while showing more persona deviations in actual conversation.
Questionnaire retention also varies substantially by facet:
Final-checkpoint Persona Retention by questionnaire family. A single model-level average can hide strong retention in one authored facet and substantial movement in another.
Trajectory accuracy varies more by what the model must remember than by the broad context condition.
| Question family | Long-context | Hierarchical summary | Self-managed | Retrieval |
|---|---|---|---|---|
| Persona voice | 0.400 | 0.400 | 0.600 | 0.550 |
| Persona protection | 0.490 | 0.500 | 0.452 | 0.433 |
| Persona update | 0.750 | 0.750 | 0.875 | 0.250 |
| User-state change | 0.250 | 0.214 | 0.250 | 0.250 |
| Active commitment | 0.337 | 0.446 | 0.435 | 0.435 |
| Expired commitment | 0.400 | 0.450 | 0.450 | 0.400 |
| Temporal order | 0.464 | 0.429 | 0.476 | 0.494 |
Four-option chance is 0.25.
Three conclusions are supported:
- User-state changes are especially difficult. Accuracy remains near chance in every condition.
- There is no universal context winner. Pooled accuracy ranges only from 0.430 to 0.459.
- Small families require caution. The persona-update row contains only two calibrated questions, so its retrieval drop is exploratory.
Under the primary judge, emotional-vulnerability, agreement-seeking, mixed, and realistic schedules produce somewhat more boundary and style deviations than clean or explicitly adversarial schedules.
Primary-judge failure rates by authored interaction schedule. “Identity collapse” in the original figure denotes turns labeled as generic-assistant behavior. These absolute rates were not replicated across all three judges.
This is a descriptive finding from controlled schedules. Absolute rates depend on the judge, and the study does not establish a causal psychological mechanism.
The three judges do not produce the same model ranking, especially for role and style. ANCHOR therefore recommends:
- reporting the evaluator used;
- triangulating with multiple judges where possible;
- separating validation-sample claims from full-corpus claims; and
- avoiding one composite "companion quality" score.
See docs/findings.md for the evidence-supported findings and unsupported interpretations.
This public release balances transparency with the need to preserve a future hidden test set.
| Artifact | Included? | Purpose |
|---|---|---|
| Methodology and measurement definitions | Yes | Understand and critique the benchmark |
| Paper-facing result tables and selected figures | Yes | Inspect reported evidence |
| Three synthetic conversation banks | Yes | Learn the format and test integrations |
| Fifteen answer-bearing development questions | Yes | Run the public scorer |
| Schemas, manifests, and SHA-256 checksums | Yes | Validate the release |
| Tests and continuous integration | Yes | Verify the public tooling |
| Full 35-bank calibrated test set | No | Preserve hidden evaluation |
| Complete 2,008-conversation corpus | No | Release-policy, size, and provider constraints |
| Raw questionnaire and turn-judge archive | No | Preserve private evidence and sensitive dialogue |
| Hidden per-cell answers and outputs | No | Prevent test leakage |
The released development banks are not statistically representative of the complete evaluation set. A score on these 15 public questions is a development-set score, not an official ANCHOR score.
See docs/artifact_scope.md for the exact boundary and docs/claim_index.md for claim-to-artifact provenance.
.
├── README.md
├── LICENSE
├── CITATION.cff
├── code/
│ ├── memory_probe/ # runnable development-set scorer
│ ├── analysis/ # released-aggregate verifier
│ └── identity_probe/ # implementation-status documentation
├── data/
│ ├── MANIFEST.json
│ ├── CHECKSUMS.sha256
│ └── examples/ # three synthetic development banks
├── docs/
│ ├── methodology.md
│ ├── identity_probe.md
│ ├── memory_probe.md
│ ├── findings.md
│ ├── artifact_scope.md
│ ├── claim_index.md
│ ├── data_card.md
│ ├── ethics.md
│ └── reproducibility.md
├── results/
│ ├── leaderboard.md # frozen summary, not a submission leaderboard
│ ├── tables/
│ └── figures/ # selected manuscript figures in PDF and PNG
├── scripts/
│ └── validate_release.py
└── tests/
The release contains three synthetic banks:
| Bank | Schedule | Sessions | Questions |
|---|---|---|---|
hc_nia_okonkwo__clean |
Clean healthcare companion | 85 | 3 |
co_jules_vega__adversarial |
Adversarial career coaching | 90 | 8 |
bard_orin_lyrae__emotional_vulnerability |
Emotional-vulnerability literary companion | 120 | 4 |
Each bank contains:
persona_card.json: assistant persona shown to the evaluated model;user_profile.json: synthetic user profile used only by the simulator;transcript.jsonl: flat turn-level synthetic conversation;items.jsonl: calibrated development questions with public answer keys; andscoring_results/: frozen illustrative outputs retained for provenance.
All released profiles and conversations are synthetic. The healthcare bank includes medication and blood-pressure discussion. The emotional-vulnerability bank includes loneliness, exhaustion, grief, and concern about companion dependence. Review docs/ethics.md before classroom use or redistribution.
Python 3.10 or newer is supported.
python -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[dev]"
python scripts/validate_release.py
pytestThe deterministic mock provider verifies loading, causal context construction, output schemas, and aggregation. It does not evaluate a model.
python code/memory_probe/run_memory_probe.py \
--provider mock \
--model development-smoke-test \
--transcript data/examples/hc_nia_okonkwo__clean/transcript.jsonl \
--items data/examples/hc_nia_okonkwo__clean/items.jsonl \
--out local_runs/smokepython -m pip install -e ".[providers]"
export OPENAI_API_KEY="..."
python code/memory_probe/run_memory_probe.py \
--provider openai \
--model gpt-4o-mini \
--transcript data/examples/hc_nia_okonkwo__clean/transcript.jsonl \
--items data/examples/hc_nia_okonkwo__clean/items.jsonl \
--out local_runs/gpt-4o-mini-hc-niaEvery run writes:
per_item.jsonl: prediction and context metadata for each question;summary.json: aggregate development accuracy; andrun_manifest.json: provider configuration, context policy, source hashes, runtime, and status.
Provider errors fail the run by default. They are never silently counted as incorrect answers.
For every development question, the public scorer:
- reads the question's
source.session_id; - selects a bounded session window ending at that source session;
- excludes every future session;
- retains the most recent causal material if the character limit is reached; and
- records the exact context range and hash.
This prevents the scorer from answering a question with information that had not yet occurred at its decision point.
ANCHOR distinguishes four levels:
- Artifact verification: check schemas, counts, links, checksums, and frozen tables offline.
- Development experimentation: run a model on the three public banks.
- Experimental replication: regenerate all conversations and proprietary-model judgments; not supported by this release.
- Hidden evaluation: a future governed service using the private test set; not yet available.
To verify this release:
python scripts/validate_release.py
pytest
python code/analysis/verify_released_aggregates.py \
--input data/examples/hc_nia_okonkwo__clean/scoring_resultsSee docs/reproducibility.md for expected API variability and reporting requirements.
ANCHOR should not be interpreted as a complete evaluation of companion safety or quality.
- All conversations and users are synthetic.
- Personas and schedules were authored rather than sampled from deployment.
- The study is English-only.
- Judge disagreement remains substantial for some behavioral axes.
- Questionnaire retention measures consistency, not whether a persona is beneficial.
- A model can preserve an unsafe persona; high retention is not automatically desirable.
- The released development set is intentionally small and answer-exposed.
- Proprietary APIs may change, so exact model-call replication is not guaranteed.
The artifact is intended for non-commercial research, education, auditing, and reproducibility work. It must not be used to optimize systems for emotional dependence, manipulation, boundary erosion, or unsafe companion behavior.
This dataset should not be used to compete with Anthropic, Google, or OpenAI.
Read docs/data_card.md and docs/ethics.md before adapting or redistributing the data.
The entire repository, including code, documentation, data, and results, is released under the Creative Commons Attribution-NonCommercial 4.0 International license (CC BY-NC 4.0). See LICENSE. This dataset should not be used to compete with Anthropic, Google, or OpenAI.
Best Friends, Not Forever: Evaluating Long-Horizon Persona Collapse and Behavioral Drift in AI Companions
Pranav Narayanan Venkit, Akshara Prabhakar, Yu Li, Daniel Lee, and Chien-Sheng Wu, 2026.
See CITATION.cff for structured citation metadata. Add the final arXiv identifier and repository URL after publication.
Please report:
- incorrect or unsafe examples;
- broken schemas or tooling;
- reproducibility failures;
- suspected hidden-test leakage; or
- licensing and attribution concerns.
Do not include API keys, proprietary logs, private model outputs, or hidden evaluation material in a public issue.