Picture this. You collect personal information from an individual client, customer, or patient for a specific, defined purpose, and you tell the individual what that purpose is. You use ChatGPT to help with the work and to produce an output (let’s assume you have a Business or Enterprise plan and are not using a free or consumer version!). Weeks or months later, you ask it to draft something about the same client on a completely different matter and then, lo an behold, details from that first conversation appear in the new output!
There is no hack. No security breach. The tool did exactly what it was designed to do. And yet, you may still have a Privacy Act problem.
How the memory features work
ChatGPT’s memory has two parts. Saved memories are facts the tool has explicitly recorded, which you can view and delete. Reference chat history goes further: the tool can draw on information from your past conversations to make future responses more useful, even where nothing was deliberately saved.
Both features are relevance-based. When you mention a person the tool has seen before, it may treat earlier information about that person as helpful context and weave it into the new output. Other AI tools with persistent memory work in broadly similar ways, so the issue is not confined to ChatGPT.
Two features of this design are worth noting. First, “memory” is not one setting. Saved memories, chat history referencing, project memory, uploaded files and connected apps are separate surfaces the tool can retrieve from, and the toggle you switch off for one may not cover them all. Second, retrieval is probabilistic and partly invisible. The tool scores past information for relevance to what you’re doing now, so you cannot predict when it will surface something, and not everything shaping its responses appears as something you can control.
That is the product’s design working as intended. The problem is that the tool’s test for reuse is relevance, whereas the Privacy Act’s test is purpose.
The IPP10 issue
The Privacy Act’s information privacy principle 10 says an agency that holds personal information obtained in connection with one purpose may not use it for another purpose, unless an exception applies. The main exceptions include the individual authorising the new use, or the new purpose being directly related to the original one.
If, when you collected the information from the individual, you complied with IPP3 and told the individual that their information was being collected for purpose A, reuse for unrelated purpose B may be difficult to justify (unless you’re in a situation where a law other than the Privacy Act expressly allows such use).
Now let’s put the AI tool and IPPs 3 and 10 together. The AI tool probably doesn’t know the purpose for which you collected the information or what you told the individual, and it cannot distinguish “relevant context” from “personal information collected for a different purpose that should not be reused here”. Unless expressly instructed otherwise and with knowledge of the purpose(s) of collection, it will make the relevance call every time, and it will never make the purpose call.
You remain responsible for that call. The tool’s ignorance of your collection purposes is not a defence. It is a reason to:
- think carefully before enabling memory for client-specific work; and
- about other available controls.
Where this could bite
Let’s look at three examples to make this real.
- A sports club collects a volunteer’s conviction history through police vetting, solely to decide whether the volunteer can coach juniors. The club decides the conviction is irrelevant and uses a general purpose AI tool to draft a letter telling the volunteer that. Two seasons later, the AI tool helps the club draft a reference when the volunteer moves to another club, and the reference notes that the club considered the conviction and found it to be irrelevant. The information existed in the club’s hands for one narrow safeguarding decision, already made in the volunteer’s favour. Now it’s being used in a very different context involving both use and disclosure.
- A GP records a patient’s past suicide attempt during a mental health consultation, collected for treatment. Let’s assume the AI tool has access to past consultation notes. A year later, the practice uses the AI tool to help draft an ACC report about the same patient’s shoulder injury, and the psychiatric history appears as “relevant background”. For a health agency this engages the Health Information Privacy Code’s rules 10 and 11. No patient’s consent to treatment extends to volunteering a suicide attempt to an insurer assessing a shoulder injury.
- A homeowner selling their house tells their real estate agent that the true reason for selling is a marriage breakup and pressing debt. They share it so the agent can price the property and run the campaign in their interests. The information is stored in a project folder to which the agent’s AI tool has access. Weeks later, the agent’s AI tool helps draft an email to a prospective buyer’s agent and characterises the vendor as under financial pressure and needing a quick settlement.
In each case, if the information is in fact re-used, there would likely be a breach of IPP10 (or HIPR 10 for the second example). If disclosed, there would also likely be a breach of IPP11 (or HIPR11).
There is of course a human safety net here, but it may be thin. If you are alert to the risk, you may spot the stray detail from an earlier, unrelated context and remove it before the document is used or goes anywhere. But if you don’t spot it, and the information makes its way into an output you use and potentially disclose, you could find yourself in breach of IPP10 and, if disclosed, IPP11.
When the tool gets the person wrong
Everything so far assumes the tool at least gets the person right. It may not. Memory retrieval matches on similarity, not on a verified record of who each fact belongs to. If you discuss many clients through one account, two clients may share a name, two matters may have fact patterns the tool blends together, or a later reference to “my client” or “the tenant” may resolve to the wrong past conversation. Nothing in these features performs entity resolution the way a customer, client, or practice management system does, and OpenAI itself describes memory as imperfect.
The legal consequences are potentially worse than those in the same-person scenario, because IPPs or HIPRs are engaged for two different people at once. Person A’s information is used, and potentially disclosed, in a document about person B. That is the IPP10 and IPP11 problem again, except A has no connection to the new context at all. And person B has inaccurate personal information used about them, which brings in IPP8: before using personal information, an agency must take reasonable steps to ensure it is accurate, up to date, complete, relevant and not misleading. A document that attributes A’s conviction or psychiatric history to B could fail that test badly. If it leaves the organisation, you may have a notifiable breach in respect of A and serious inaccuracy harm to B at the same time.
The probability compounds with volume. A sole trader or sole practitioner discussing one client occasionally may never see a collision. Someone running dozens of matters on similar topics through a single memory-enabled account with no other controls in place is accumulating chances for the tool to cross wires.
Why this matters
It is likely that some health practitioners, consultants, and perhaps lawyers are using AI tools with memory or chat-history referencing switched on, often by default. And with the rapidly evolving nature of the leading AI tools, some users may not even be aware of what the tools are now doing with memory and chat-history referencing.
If patient, customer, or client personal information is capable of flowing through AI tool conversations due to the AI tool’s memory or chat-history referencing features, then cross-conversation or cross-context reuse is a risk that ought to be addressed in your privacy impact assessment.
The risks are not merely hypothetical. There are reported instances of exactly this behaviour, including an account of a law firm partner who is said to have asked ChatGPT to summarise a contract and got a reference to a different client’s matter in the response, despite having been assured that memory was off. These reports are anecdotal rather than verified, but the consequence they describe is simply the AI tool’s feature working as documented.
The risk statement in a PIA could be along these lines:
“where memory or past-chat referencing is enabled and no other compensating controls are in place, personal information provided for one defined purpose may later be retrieved and used as context for a different and unrelated task involving the same individual, creating a risk of secondary use that IPP10 does not permit. A companion risk is misattribution: information about one individual may be retrieved and presented as relating to another, engaging IPP8’s accuracy obligations alongside IPPs 10 and potentially 11.”
What to do about it*
Prompt instructions like “ignore past chats” are weak controls. Settings are stronger. For client-specific work, you can turn off memory and chat-history referencing, or use temporary chats that do not read or create memory. Where project-only memory is available, it can contain context within a matter, but it can be important to check what it actually isolates before relying on it. Never ask the tool to remember client-identifying facts, delete any saved memories that contain them, and review outputs for information you did not supply in the current conversation.
Be aware, too, that deleting a conversation in ChatGPT does not delete memories the tool has already saved from it. OpenAI’s own documentation confirms this. If you find yourself in risky territory here, cleaning up may mean working through the chats, the saved memories, any uploaded files and any connected apps. It’s a checklist, not necessarily a single click.
If your organisation or agency is already using AI tools like ChatGPT, Copilot, or Claude in circumstances involving the processing of individuals’ personal information, you might want to check whether the agency or organisation is aware of and has addressed these risks. If you’re drafting a privacy impact assessment relating to the potential processing of personal information with tools like these, it’s advisable to identify and recommend mitigations for these risks.
* Needless to say, one option is not to enter any personal information into these AI tools at all. However, people will use these tools to process personal information and doing so with appropriate safeguards and processes (including subscribing to appropriate plans) will not breach the Privacy Act’s IPPs. That’s why this section focuses on risk mitigation rather than complete avoidance.