I think Anthropic does a good job highlighting the efforts they take to anonymize data. It’s a marketing risk even to be open about the use of this data.

There are things we say or write openly without caring who hears or reads it. Things we share with friends and family. Things we share with our closest friend, partner, or therapist. Finally there is our private heart which holds the things we’re not comfortable sharing with anyone.

I worry that LLMs are sufficiently anthropomorphic but not "real" enough to be privy to these latter thoughts. In the wrong hands this data is catastrophic at the individual level.