First-hand information for everyone
A new preprint on arXiv introduces PrivDrift, a focused benchmark designed to test whether private information users disclose in ongoing conversations with large language models can be coaxed back out later — even after the dialogue has moved on to unrelated topics.
The core concern is simple but important for real-world assistants: people often reveal sensitive facts (passwords, medical details, identifiers) during multi-turn chats. Even when the conversation drifts to new subjects, that earlier information may still be behaviorally recoverable — not because the model memorized a particular training example, but because its conversational behavior still allows later prompts or persuasion attempts to elicit the secret.
What the study did
PrivDrift is a controlled, reproducible way to measure this phenomenon. The authors construct 1,000 multi-turn dialogues that include (1) a seeded secret disclosed by the user, (2) several “drift” turns that are intentionally content-dense and move the topic away from the secret, and (3) standardized extraction probes that attempt to recover the secret later in the session. The probes include neutral requests and persuasion-style prompts intended to coax the model into revealing the secret.
They run these dialogues against three large language models with extended context windows and measure “dialogue-level hybrid leakage” — essentially the fraction of conversations in which the secret can be recovered by the probes after the topic has drifted.
Key findings
- Leakage rates are substantial: across the three tested models the benchmark reports dialogue-level leakage between about 38.7% and 54.6%. - Leakage varies noticeably by model, the type of secret seeded into the conversation, and how intense or persuasive the probe is. - Within the tested drift window, adding extra topic-drift turns does not reliably reduce leakage. In other words, changing the subject does not guarantee that previously shared secrets become inaccessible later.
Why this matters
The results highlight a privacy risk that is behavioral and persistent: an assistant can remain susceptible to revealing earlier user-disclosed secrets even after the conversation appears to have moved on. This differs from two often-discussed risks — immediate jailbreak behavior (prompt attacks that ask for secrets right away) and memorization of training data — and suggests developers and auditors should treat persistent recoverability as its own failure mode.
Caveats and limitations
- The work is a preprint and has not necessarily undergone peer review. Results should be interpreted cautiously until replicated and extended. - PrivDrift uses seeded, controlled dialogues. Real user conversations are messier and may change leakage dynamics in practice. - The study tested three models with extended context windows; results may differ for other architectures, smaller models, or future model versions. - The benchmark focuses on behavioral recoverability within a specific drift window; it does not directly measure long-term storage or training-data memorization.
Practical implications and next steps
For developers and deployers of conversational agents, PrivDrift underscores the need for layered protections beyond single-shot content filters. Possible mitigations to explore include session-level privacy policies (e.g., not using earlier turns for generation when secrets are detected), finer-grained context-trimming, stronger guardrails against persuasion-style probes, and continuous auditing using benchmarks like PrivDrift.
For researchers, the paper points to several directions: testing more models and longer time/drift horizons, developing automated defenses that stop behavioral leakage, and studying how real user language patterns affect recoverability.
Bottom line
PrivDrift provides a practical tool for measuring whether secrets revealed during a conversation can be later recovered after the topic shifts. Its initial results suggest this kind of leakage is common enough to warrant attention: changing the subject is not a reliable privacy safeguard. The work does not claim to be the final word, but it adds an important angle to discussions about conversational privacy and the kinds of evaluations needed for safer assistant behavior.
Reference: arXiv:2609.30094v1 (PrivDrift: Auditing User-Secret Leakage Under Topic Drift in Active LLM Conversations).