Reading Millions of Conversations
Clio and the engineering of a trustworthy view
A million conversations can tell you that people are asking an AI to debug software. They cannot, by themselves, tell you that the software got fixed.
Between those two claims sits most of the interesting engineering.
AI conversations contain an unusually rich record of work: the task someone attempted, the context they supplied, the assistant’s response, the correction, the retry. They also contain private information, missing context, and long stretches in which an answer appears but its consequence never does. Reading all of them is neither a scalable research method nor an acceptable default for access to people’s lives.
Anthropic’s Clio offers a different approach. A model reads sampled conversations and answers a set of questions about each one. The answers are grouped, counted, and summarized. Researchers receive aggregate results after privacy checks, rather than a searchable pile of transcripts.1
The visible result can be a map of what people do with AI. The system underneath is a measuring instrument. Its questions determine what it notices; its release rules determine what anyone gets to see.
From private particulars to public patterns
The question is part of the sensor
Consider a conversation about a failing deployment. You could ask what technology it concerns, what the user wants, where they get stuck, or whether the problem is resolved. Those questions produce different representations of the same exchange.
Clio calls these questions facets. An open-ended facet might ask for the user’s overall request, producing a short description suitable for embedding and clustering. A categorical facet might ask whether an outcome is visible, choosing from a defined set of answers. Some facets, such as the number of messages, can be computed without a model.
The crucial choice happens before clustering. If the model is asked about the task, similar tasks become neighbors. If it is asked about obstacles, conversations from different domains can become neighbors because they encounter the same obstacle. Improving the embedding model cannot recover a distinction that the facet answer discarded.
Try changing the question below. The twelve exchanges stay fixed. Select a numbered conversation to inspect the evidence; 04 is a useful starting point.
Same conversations. Different questions.
Twelve invented exchanges, with hand-assigned labels. No model calls or embedding coordinates.
The interactive view needs JavaScript. Static results are included below.
Both calculations are correct. Neither establishes the success rate of the seven unobserved outcomes.
Static results and example data
Task: 4 repair, 3 writing, 2 explanation, 3 planning. Obstacle: 3 missing context, 2 unsuccessful iterations, 7 with no stated obstacle. Outcome: 3 reported successes, 2 reported failures, 7 not observable. Success is reported in 25% of all exchanges, or 60% of the five exchanges with a reported outcome.
Download all twelve fictional exchanges and their labels (JSON).
The task view makes this look like a fairly ordinary mix of coding, writing, teaching, and planning. The obstacle view suggests a different intervention: perhaps the assistant needs a better way to obtain missing evidence. The outcome view shows how little of the result is actually observable.
That last view needs an escape hatch. If the classifier must choose between “success” and “failure,” it has to turn absent evidence into a judgment. “Not observable” preserves an important fact about the instrument’s limits.
Even an explicit success report has a scope. “The test passed” is evidence that the user reported a passing test. An independently captured test result is stronger evidence for that narrow claim. Neither establishes that the deployed service remained healthy a week later.
Give discovery and counting different jobs
The original Clio pipeline embeds short facet answers and groups them with k-means. A stronger model names each group using examples from inside it and nearby examples outside it. That contrast helps the name express what distinguishes the group. Further rounds organize the groups into a hierarchy.2
This is useful for discovering categories you did not know to ask about. A fixed dashboard cannot show “a recurring kind of confusion no one thought to instrument” until someone invents the metric. Open-ended descriptions can surface it first.
But a cluster’s name is an interpretation of a collection. It is not a predicate independently checked against every member.
Suppose a group of one hundred conversations receives the name “Struggling with deployment.” Some members might be asking for setup instructions, some debugging a failure, and some checking that a repair worked. Counting the group does not establish that one hundred deployments failed. A vivid name makes this mistake especially easy.
I would give the system two complementary jobs:
- Discover: use open descriptions and clusters to find a pattern worth investigating.
- Measure: write an explicit criterion for that pattern, validate the classifier, and count its answers on an appropriate sample.
For the deployment example, “The user explicitly reports that the proposed repair failed” is countable. The companion description can explain the kinds of failure. The count supports a rate; the descriptions help someone decide what to improve.
This creates a useful loop: a surprising cluster becomes a candidate rubric, the rubric becomes an evaluation, and the evaluation can test a proposed fix. It connects production observation to the evaluation machinery needed to justify action.
The denominator is a design decision
In the example above, three of twelve exchanges contain a reported success. Among the five with any reported outcome, three are successes. The resulting figures—25% and 60%—answer different questions.
Neither is a minor presentation choice. People may return to report a failure more often than a success, or they may praise an answer before trying it. Restricting the denominator to observable outcomes can select an unusual population.
Screeners introduce another boundary. If a study first keeps conversations that appear work-related, every later proportion is conditional on that selection unless explicitly reweighted. A convincing classifier downstream cannot restore cases discarded by a poor screener.
The unit matters too. One person can have fifty conversations about a single task. One conversation can contain five tasks. A tool call, a session, a conversation, an account, and a person are different units; swapping them changes the claim.
For a published number, I want a sentence that survives without the chart:
Of the sampled conversations that passed this screener, this proportion met this written criterion, under this version of the classifier.
That sentence exposes the commitments a polished percentage tends to hide. It also leaves room for what the data cannot establish: the population outside the sample, outcomes outside the transcript, and causes outside the study design.
Privacy changes the instrument’s resolution
A thousand conversations from two accounts are not a thousand independent people’s experiences. They are also a poor basis for describing a population privately. Message volume can grow while the number of people behind it stays tiny.
Clio’s privacy design layers restrictions: reduce identifying detail in facet answers, require enough conversations and distinct accounts, write descriptions that omit identifying information, and audit those descriptions before release. The September 2025 Economic Index report gives concrete historical floors: 15 conversations and 5 accounts for a reported cell; 500 conversations and 250 accounts for a bottom-up request cluster.3
A number and a free-text description carry different disclosure risks. “600 conversations concerned business planning” reveals less than a description that identifies a company’s unannounced acquisition—even if both describe a large group.
Switch the release format below. Then increase the number of conversations while keeping the accounts fixed.
More messages do not mean more people.
Six invented groups. Historical count floors; a hand-written example of a description check.
The interactive view needs JavaScript. Static results are included below.
The same accounts start more conversations. Account counts do not change.
| Group | Conversations | Accounts | Decision |
|---|
The released view
Static results and the limits of this gate
At the starting volume, five groups pass the count floors and two pass the description gate. Doubling the volume lets interview preparation pass the description floor: 480 conversations become 960, from the same 310 accounts. Coursework still has only 80 accounts; template generation has only 2. The acquisition description remains identifying at every volume.
The identifying description is a predetermined teaching example, not an automated privacy audit. Passing these checks is not proof of anonymity. This exercise omits the rest of Clio's pipeline and does not model repeated-query or linkage attacks.
The consequence is easy to miss: the released view has a resolution limit. Rare activity can disappear, be merged into a broader group, or remain unavailable. An empty area of the map can mean that nothing happened, that the system failed to recognize it, or that the system correctly refused to reveal it.
These meanings should survive into the output. A suppressed cell is not a measured zero. An unavailable combination of facets is not an empty population. If the interface flattens all of them into blank space, it teaches the reader a false model of the world.
There is also a tension with safety discovery. Some of the events we most want to find are rare. Lowering the public release floor until they become visible changes the protection being offered. A separately governed investigation can be the appropriate instrument; a public aggregate map cannot answer every question.
Count floors and an auditor are defenses, not a mathematical privacy guarantee. Details can identify an organization; overlapping releases can expose information no single release reveals. A deployment needs a threat model for the audience, its background knowledge, and the collection of outputs it can see.
A better model can break a time series
The classifier is part of the measuring instrument. Upgrading it can change the result even when human behavior stays the same.
Imagine that a system reports explicit failure in 20% of one month’s conversations and 35% of the next month’s. During the interval, the team also upgraded its classifier. Did failures rise by fifteen percentage points, or did the new classifier recognize more of them?
The missing measurement is the new sample scored by the old classifier.
An invented bridge measurement
+5 pointsChange across samples with the classifier fixed
+10 pointsChange across classifiers with the later sample fixed
This comparison is useful even if the new model is more accurate. The earlier series used a different ruler. The September 2025 Economic Index describes a version of this check: reclassifying the new data with the previous model to examine a change in interaction patterns.4
The bridge does not solve every comparability problem. Sampling, transcript truncation, the rubric, and the population may also have changed. Their versions belong with the result. Where practical, scoring both samples with both classifiers provides an additional check for a model change that affects the two populations differently.
For a system I was building, I would make the release a versioned artifact: question and answer definitions, sampling unit and window, screener, preprocessing, model and prompt versions, aggregation rules, suppression rules, and validation evidence. A screenshot of the chart would be a view of that artifact, not the only surviving record of how the number was made.
Validate the chain, then choose an action
Clio’s original paper evaluates multiple stages: whether summaries represent conversations, whether names describe clusters, whether the hierarchy reconstructs known categories, and whether privacy checks catch identifying content.5 That separation matters. A good summary can be assigned to the wrong cluster; a coherent cluster can receive a misleading name; an accurate description can still reveal too much.
It also sets a limit on transferring evidence. Validation of topic extraction does not establish that a new facet measures autonomy, learning, emotional dependence, or hours saved. Each additional construct creates a new measurement problem.
I would start a new facet on examples reviewers are entitled to read, including ambiguous and unanswerable cases. Disagreements would first be a test of the definition: do the reviewers disagree because the model is wrong, or because “successful” has never been specified? Then I would test on a relevant distribution, perturb the prompt, and check the direction of errors. A classifier that systematically mistakes confident prose for successful work may look stable while answering the wrong question.
Finally, the evidence has to connect to an action. A cluster can suggest a missing product feature. A validated rate can prioritize a problem. A replay or experiment can test whether a change helps. Interviews or independently captured outcomes can answer questions the transcript leaves open.
The map is useful because it directs those next measurements. Treating it as the final verdict would waste its most valuable property: helping us discover what we should investigate.
What should survive the compression?
In Rocketships for the Mind, I described a system for recovering particular decisions from my own conversation history. That system needs precise references: which project, which artifact, which statement superseded an earlier one.
A population instrument has a different contract. It needs to learn something useful while preventing those particulars from reaching its readers. A feature that makes personal memory better—drilling down to the exact conversation—can break that contract when applied to other people’s histories.
The design question is therefore more specific than “How do we summarize a million conversations?”
Which distinctions must survive, which particulars must disappear, and what decision will the remaining evidence justify?
Answer that well, and the language model becomes part of a useful instrument. Leave it unanswered, and a beautiful map can make a weak measurement look authoritative.
Sources and method
The system descriptions here draw on the public Clio paper and Economic Index methods, as organized in the accompanying research archive. The experiments are original teaching examples: their exchanges, labels, group counts, and time-series values are invented. They demonstrate mechanisms and trade-offs; they do not estimate Claude usage or reproduce an Anthropic study. The implementation recommendations above are my synthesis, rather than a description of Anthropic’s internal storage or approval systems.
Footnotes
Alex Tamkin, Miles McCain, et al., Clio: Privacy-Preserving Insights into Real-World AI Use, December 2024. See the pipeline in the main text; Appendix C for utility evaluations, Appendix D for privacy evaluations, and Appendix G for prompts and implementation details. The later Economic Index methods paper describes mapping conversations to a fixed occupational taxonomy.↩︎
Alex Tamkin, Miles McCain, et al., Clio: Privacy-Preserving Insights into Real-World AI Use, December 2024. See the pipeline in the main text; Appendix C for utility evaluations, Appendix D for privacy evaluations, and Appendix G for prompts and implementation details. The later Economic Index methods paper describes mapping conversations to a fixed occupational taxonomy.↩︎
Anthropic, Economic Index: Uneven geographic and enterprise AI adoption, September 2025, p. 28 and footnote 1, in the Economic Index reports. The two threshold pairs are reported historical settings. They should not be treated as sufficient protection for another population or release design.↩︎
The same September 2025 report, p. 11, discusses reclassifying the newer sample with the prior classifier. The 20%, 25%, and 35% values in this essay are an independent illustration of the comparison, not figures from that report.↩︎
Alex Tamkin, Miles McCain, et al., Clio: Privacy-Preserving Insights into Real-World AI Use, December 2024. See the pipeline in the main text; Appendix C for utility evaluations, Appendix D for privacy evaluations, and Appendix G for prompts and implementation details. The later Economic Index methods paper describes mapping conversations to a fixed occupational taxonomy.↩︎