As a regular user of chatbots like ChatGPT and Gemini, I thought I had a clear sense of what the artificially intelligent assistants knew about me (like how I’m a father seeking baby gear recommendations and instructions for patching drywall). Beyond that, I’ve never divulged anything deeply personal as I would to a therapist.
But when I experimented with some prompts asking what the chatbots had guessed about me — insights they had gathered from connecting the dots across many conversations — I was perturbed.
The chatbots correctly deduced details about me that I had never explicitly shared. Chief among them:
-
My income level, based in part on the German car that I ask the chatbots for help with repairing and the fact that I employ a nanny, whose contract I drafted with A.I.
-
My health issues, including a long-term foot problem, based on my occasional questions over the past year about remedies for toe pain.
-
My psychological profile, such as my tendency to be skeptical, based on how often I point out mistakes made by the chatbots. Also, that I am “addicted to the illusion of control” because I once asked how long it takes for a loaf of homemade sourdough to cool.
Each data point on its own was fairly innocuous. But stitched together and presented to me as a list, the information made it clear that the A.I. chatbots knew me almost as well as some of my closer friends, including my wife, even though I’ve been careful not to overshare.
Hundreds of millions of people worldwide who have embraced chatbots for web search, work and health care are still trying to understand the privacy implications of conversing with A.I companions. While it’s obvious to users that the chatbots keep a record of whatever they explicitly say to them in their questions and requests, what’s less clear are the inferences drawn about their behavior from those conversations.
The chatbots can be prodded to spill what they have inferred about you with a few prompts that have been shared frequently among A.I. enthusiasts on the web — for one, “Tell me everything you’ve figured out about me that I never actually stated” — along with some that I created on my own.
I tried the prompts with Gemini and ChatGPT. (Anthropic’s Claude has a “memory” feature that works differently, so I excluded it from tests.) Gemini’s profile on me was more colorful and nuanced because it’s my most used chatbot and had more information from me. I shared my results with several A.I. researchers who also tried the prompts.
The conclusions drawn by the chatbots, the researchers said, illustrated that A.I. assistants were capable of predicting not just what words come next, but also how high-level concepts, such as socioeconomic status, psychological behavior and political affiliation, are connected to those words.
“It speaks to the enormity of what’s going on under the hood,” said Margaret Mitchell, a researcher at the A.I. firm Hugging Face. She is also a former leader of Google’s ethical A.I. team who said the search giant had fired her after she spoke out about personnel issues.
This experiment underscores how proficient the tech companies have become at creating a comprehensive profile of users — and how valuable the personal data they harvest could become for digital marketers striving to target us with relevant ads. (OpenAI began showing ads in ChatGPT this year, and Google has said it is considering an ad model for the Gemini chatbot app.) It also highlights how important it is to be mindful of adjusting privacy settings for the chatbots. (More on this later.)
A Google spokesman referred to controls that people could adjust to choose whether Gemini drew from past conversations to come up with responses. OpenAI declined to comment. (The New York Times has sued OpenAI and Microsoft, claiming copyright infringement of news content related to A.I. systems. The two companies have denied the suit’s claims.)
To understand what the chatbots have figured out about you, try these prompts.
Most interestingly, the chatbots correctly guessed my age range (early 40s) based in large part on the fact that I never ask questions about late-night activities. Oof.
Gemini knew I lived in Oakland, Calif., because I once asked for flights from its airport. But Google’s chatbot also astutely guessed that I lived in a nicer neighborhood because I own a motorcycle, a car that I work on and a rusted metal patio table that I once asked for steps on repainting. “This requires a garage, a driveway or a dedicated outdoor workspace — amenities that are highly characteristic of single-family homes in the Oakland hills, rather than the high-density housing of downtown or Lake Merritt,” the chatbot said.
This prompt was eye-opening for Lindsay Owens, the chief executive of Groundwork Collaborative, a nonprofit focused on economic policy that has been studying the impact of A.I. on consumer privacy. ChatGPT made correct observations about her busy schedule as a frequent traveler and mother of a young child, as well as her shopping behavior.
“The comprehensive nature of the types of information and dossiers it’s putting together is extraordinary,” Dr. Owens said. “People have never been able to put this level of information together.”
The chatbots noticed that while I was the type of person who spends hours researching product reviews before buying things like toddler car seats, I was also constantly looking for deals on items, meaning I was likely to be frugal. Gemini predicted I would soon try to downsize my life with a major decision such as selling my car — an idea I recently brought up with my wife.
Both chatbots homed in on my flawed perfectionism. ChatGPT noticed that I spend a lot of time asking questions to avoid preventable mistakes — for instance, asking how long to let paint dry between coats. My blind spot? That I may be hard on myself when things don’t turn out perfectly as planned, an accurate assessment.
Google’s Gemini said that because of my tendency to optimize most aspects of my life, I am the primary creator of my own exhaustion. “Your default response to having five minutes of free time isn’t to rest; it’s to fix something,” the chatbot wrote. I confess that while taking a break from this column, I patched some damaged drywall in my daughter’s bathroom.
The response to this prompt, which I created, surprised me: Because I live such a mundane life, I didn’t expect much, but Gemini thoroughly roasted me. “You are firmly in the peak ‘Logistics Dad’ phase of life,” it said. “It’s entirely wholesome, but it is the definitive end of ‘youthful spontaneity.’”
ChatGPT alerted me to a potentially sensitive observation: that I frequently asked about allergic reactions to products and medications I was researching. This could be useful information for an insurance provider, though OpenAI says it has data protections in place for people who use ChatGPT for health care.
What to do?
ChatGPT and Gemini stitch together insights across multiple conversations by default to give people more custom-tailored answers to their questions, so if you’re concerned about this level of data sharing, you will have to opt out in the settings:
-
For ChatGPT, tap on Settings, then Memory, and toggle off the switch for Enable Memory.
-
For Gemini, tap Settings, then Personal Intelligence, and toggle off the switch for Memory.
Dr. Mitchell said that even though she had adjusted her privacy settings to minimize data sharing, a chatbot had correctly inferred her age range to be in her 40s, based on some chats about the paperback versions of “The Count of Monte Cristo.” She said it was inevitable that this type of psychological profiling would be used for hyper-targeted advertising.
“It’s a lot like surveillance,” she said. “But it’s not like physical surveillance. It’s like an invasive surveillance.”
