"Oops ... that's not my conversation!"
On March 20, 2023, users of ChatGPT that other people's conversation titles suddenly appeared in their chat history. OpenAI temporarily deactivated the service and ultimately blamed a Redis bug for the disclosure of personal information.1 2 3
The incident is an example of how quickly confidential data can escape.
When training data becomes a threat to your data
Generative AI is fed with huge amounts of data, which is often obtained via web scraping. This includes - intentionally or not - personal or copyright-protected content. A prominent example: the ongoing lawsuit by Getty Images against Stability AI. At its core is the allegation that the Stable Diffusion image model was trained with millions of copyrighted photos, which can be seen, among other things, in the Getty watermarks visible in the output.4
Where personal data can end up in the model
As early as 2020, a study by Carlini et al. showed that GPT-2 can output personal information such as names or telephone numbers word-for-word, provided they were included in the training data.5 The potential risk increases with the size of the model and the amount of unfiltered input data.
Data leaks in AI - how confidential information escapes.
A data leak is the disclosure of confidential data caused by a software error or malicious action.
Attack vector 1 - Accidental output:
Prompt injection or simple queries can lead to the model inadvertently revealing content from the training data.
Attack vector 2 - Reconstruction:
Researchers demonstrate methods with which data (e.g. patient diagnoses) can be reconstructed from a fine-tuned LLM, even if they are not output directly.
Attack vector 3 - infrastructure bugs:
The Redis bug mentioned at the beginning is an example of how the operating environment of an AI service can also become a risk.
Between GDPR and EU AI Act: what matters for AI now
Since July 2024, the EU Artificial Intelligence Act (AI Act) was passed and will become applicable in stages (edpb.europa.eu). For high-risk systems - including many GenAI applications with personal data - it prescribes strict requirements for data governance, transparency and security engineering. At the same time, the GDPR remains fully valid with its principles of "privacy by design" and "purpose limitation".
These 6 steps make AI safe: practical tips for companies
| Field of action | Approach | Quick-Win |
|---|---|---|
| Data inventory & Classification | Clearly mark sensitive vs. public data; Treat training and prompt data separately | Data loss prevention Rules for copy & paste in prompt windows |
| Anonymization & Synthesization | Filter out personal references before data enters the model | Off-the-shelf Use Anonymizer as pre-processing |
| Retrieval Augmented Generation (RAG) | Company knowledge remains in a separate, encrypted vector database instead of in the model itself | LangChain/LLamaIndex blueprint with access token |
| Fine-tuning under Differential Privacy | Noise during gradient updates reduces memorization | DP-SGD libraries such as Opacus |
| Red-teaming & Prompt injection tests | Simulate attacks, recreate leaks | "Purple Team Day" every 3 months |
| Reaction & Compliance | Establish reporting and deletion processes for AI outputs; DPIA for high-risk apps | One-Pager Incident-Runbook |
Conclusion: AI is not a risk - if you think about data protection from the outset
The use of GenAI is not a free pass, it only shifts the tension between innovation and data security. Companies that implement clear data governance, technical protection measures and regular audits at an early stage can exploit the potential of AI without losing the trust of their customers.
Use the full potential of AI - but safely.
