Personal data in AI systems: training sets, prompts, logs and automated decisions
An AI feature rarely arrives as a privacy decision. It arrives as a pilot, a plug-in or a button, and the personal data it touches is found afterwards. The fix is to know the entry points in advance.
Key points
- Personal data enters an AI system at six points: the data used to train or fine-tune a model, the prompts people type, the documents the system retrieves, the logs it keeps, the outputs it produces, and any decision made from them.
- Staff paste personal data into prompts because it is the fastest way to get an answer, so the control is a clear rule, an approved tool with known retention, and a way to see which tools are in use.
- Logs and stored conversations are personal data stores like any other, so they need an owner, a retention period and a way to find and delete any individual’s entries.
- The EU AI Act, Regulation (EU) 2024/1689, is a separate instrument from the GDPR, and an AI system that uses personal data can be subject to both.
Six entry points, in the order data travels
Personal data enters an AI system at six points: the data used to train or fine-tune a model, the prompts people type, the documents the system retrieves, the logs it keeps, the outputs it produces, and any decision made from them. They are listed here in the order that data usually moves through a system, from before the model exists to after it answers.
Training and fine-tuning data
Training and fine-tuning data needs the same questions as any other use of personal data: where it came from, on what basis it is used, how long it is kept and whether a person’s request can reach it. For a model you train or tune yourself, those questions are yours to answer. For a model you buy, ask the vendor what it says about the data it was trained on, and whether your data is used to train or improve its models, in writing, in the contract and not only on a web page.
Regulators have started to address the model itself. The European Data Protection Board adopted an opinion on data protection aspects of AI models on 18 December 2024, at the request of the Irish supervisory authority; read the opinion itself, and ask your counsel how it applies, before relying on a summary. The security side of tuning is in fine-tuning and training data security.
Prompts
Staff paste personal data into prompts because it is the fastest way to get an answer, so the control is a clear rule, an approved tool with known retention, and a way to see which tools are in use.
A rule that bans personal data in every prompt is usually ignored. A more workable one says which tools are approved, which categories of data may go into which tool, and what is never permitted, such as health details or children’s data in a general-purpose assistant. Finding the tools already in use is covered in finding shadow AI.
Retrieved documents
A retrieval system answers from your documents, so it can say what those documents say about a person. Two questions follow. Is the person asking allowed to see the document the answer came from? And if a person asks for their data or its deletion, can you find the copies in the search index and the embeddings, not only in the source system?
The first is an access-control question covered in RAG tenant isolation. The second is a request-handling question, and the index belongs on the data map.
Logs, outputs and decisions
Logs and stored conversations are personal data stores like any other, so they need an owner, a retention period and a way to find and delete any individual’s entries. Conversation history, prompt logs kept for debugging and evaluation datasets built from real traffic all qualify. Decide how long they are kept before the feature launches.
Outputs can create or imply personal data: a summary of a customer, a score, a guess about a person’s characteristics. Treat them as data you hold about the person, with the accuracy and request obligations that follow.
Article 22 of the GDPR concerns decisions based solely on automated processing that produce legal or similarly significant effects on a person, and a system that scores, ranks or screens people should be checked against it. The text is on EUR-Lex. A human who rubber-stamps the output may not change the answer, so the check looks at what the person can actually change. The UK regulator’s guidance on AI and data protection covers Article 22 within its chapter on fairness; the page notes the guidance was updated on 15 March 2023 and is under review following the Data (Use and Access) Act, so check its current status.
The AI Act alongside the GDPR
The EU AI Act, Regulation (EU) 2024/1689, is a separate instrument from the GDPR, and an AI system that uses personal data can be subject to both. It is published on EUR-Lex. We do not state its application dates here, because they change; the current text is checked at the start of each engagement. A plain-language article is the EU AI Act for Canadian suppliers.
The practical point is that one assessment can serve both. A system that profiles or decides about people needs an impact assessment; what you write for it is also much of what an AI governance review asks for. See privacy impact assessments.
A first-week checklist
- List every AI feature and tool in use, approved or not.
- For each, mark which of the six entry points applies.
- Record the vendor’s commitments on retention and on training with your data.
- Add logs, indexes and evaluation sets to the data map.
- Name the features that rank, score or screen people, and send those to an impact assessment.
The service is described on privacy for AI systems. For the security side, see AI security.
How the work is bounded
The scope is agreed in writing before work starts, and the engagement is quoted in writing with it.
SecHB does not issue certifications, attestations or audit opinions: those come from accredited certification bodies, CPA firms and QSAs. The work here is what an organization does to be ready for them.
Nothing here is legal advice. Where a question turns on the law, the work is done alongside the client’s counsel, not instead of them.
Questions we are asked
Where does personal data enter an AI system?
Personal data enters an AI system at six points: the data used to train or fine-tune a model, the prompts people type, the documents the system retrieves, the logs it keeps, the outputs it produces, and any decision made from them.
How do we handle personal data in prompts?
Staff paste personal data into prompts because it is the fastest way to get an answer, so the control is a clear rule, an approved tool with known retention, and a way to see which tools are in use.
Are AI logs personal data?
Logs and stored conversations are personal data stores like any other, so they need an owner, a retention period and a way to find and delete any individual’s entries.
What does the GDPR say about automated decisions?
Article 22 of the GDPR concerns decisions based solely on automated processing that produce legal or similarly significant effects on a person, and a system that scores, ranks or screens people should be checked against it.
Does the EU AI Act replace the GDPR?
The EU AI Act, Regulation (EU) 2024/1689, is a separate instrument from the GDPR, and an AI system that uses personal data can be subject to both.
Map the entry points
Name the AI features you run or plan and what data they see. The reply says what a review would cover. The service is privacy for AI systems.