// TOM'S HARDWARE US — HARDWARE & GADGET
ChatGPT transcripts are reportedly read by humans to improve responses, including those with personal information — 'Project Lilly' has seen OpenAI hire hundreds of contractors to manually review logs
When you purchase through links on our site, we may earn an affiliate commission. Here’s how it works.
AI companies don't have a great track record in areas like copyright or user privacy — unless they're the ones on the short end of the stick, that is — but it's generally known that the chat logs from platforms like ChatGPT are used for improving models. The mechanism as to how this happens was still a mystery until today. 404 Media just published a report about OpenAI's process of human review for chat transcripts, explaining how the review process works, and how it involves other humans sometimes reading private information.
The rating project's name at OpenAI is Project Lily. The publication got information on the project's instruction guides, Slack channels, real ChatGPT conversations, and, of course, the rating system to classify conversations. The operators are called "prompt reviewers," and their job is fairly simple: look at anonymized real-world chats, and judge the quality of ChatGPT's responses to assess whether they actually answer the question, and that the text doesn't overuse "AI-speak," patronizing tones, emojis, or sycophancy, among other parameters. Anthropomorphizing and stating "personal" experiences are both off the table, meaning that while it's OK for ChatGPT to say "I found some information," it's not OK for it to say "as a chef, I like to..." or "I know what that's like."
The work is "very rote," according to a reviewer, but at reportedly over $50 an hour, it's a high rate for what looks like reasonably simple work. The reviewer also said that their guidelines keep changing and are often self-contradictory, a feeling most software developers should easily identify with.
The person doesn't think that most users are aware their chats are being read by others, though, something that's particularly troubling when many use ChatGPT as an impromptu friend or therapist and put deep secrets in words for the bot to read.
While the chats allegedly go through an anonymization pass and reviewers don't see usernames, OpenAI admitted to 404 Media that the filtering may let some personal data through, especially in shorter chats. The site notes that in many conversations, the user asks ChatGPT to keep the contents secret, as well. The version of the chat handed to reviewers also reportedly includes a "user memories summary," containing a summary of the users' questions and interests, context, and potentially even location.
Crucially, Project Lily does not grade the chats' actual factual accuracy other than flagging obvious mistakes, implying that there's likely at least one more team (or several) doing separate evaluations. Likewise, this reviewing is separate from manual safety checks that ascertain if someone might be looking to hurt someone else (or, presumably, themselves).
The existence of the project also indicates that contrary to these image AI companies try to cultivate, the models don't improve just with technological advancement and better training sets — it appears you still need more than a few competent humans in the mix.
Get Tom's Hardware's best news and in-depth reviews, straight to your inbox.
By now you may be wondering about the "allow us to use your chats to improve our product" (paraphrased) setting present in most consumer-facing chat bots. That setting is turned on by default in every bot we can think of, even with many paid plans. In ChatGPT's case, it does default to off in Enterprise, Business, and Educational customers.