A story about an AI tool
This is a story about an AI tool used widely in the health sector, and its terms. At the same time, it’s a story that’s not unique to this AI tool and its provider, and so we’re going to tell the story without expressly naming the company and its product. Instead, we’re going to refer to the company as “ScribeCo” and to its product as “ScribeTool.” These are fictitious names, but what follows is drawn from the real terms of a real product.
ScribeTool is, by many accounts, an excellent product. It is an AI scribe tool designed for mental health clinicians: press record during a therapy session, and the tool transcribes and generates structured clinical notes. It is reportedly well built, widely used, and popular with practitioners. None of what follows is a criticism of the product itself. It is, however, a cautionary tale about what can lurk in the fine print.
What the FAQ says
ScribeTools’s security FAQs address data use directly. They ask: “Is my data used to train AI?” The answer is short and reassuring: “No. Your data is not used by us or any third party LLM to train AI models.”
For a clinician evaluating the tool, that’s the right answer. Mental health session recordings are among the most sensitive categories of personal information that exist.
A clear, unqualified “no” on the training question might satisfy many professionals doing their due diligence. But the FAQs do not reveal the full picture. To see the full picture of how an AI tool provider handles input and output data, or at least to obtain a fuller picture, we often need to read its terms of use and privacy policy.
What the privacy policy says
In this story, around halfway into the privacy policy and under the heading “Your Data Choices,” sits this sentence: “[ScribeCo] uses anonymous data to improve our service.”
That appears to be the default. It is not, it seems, something you opt into.
Confusingly, immediately after the sentence quoted above, the privacy policy states:
“However, if you would prefer your data to not be used for these purposes, you can use [ScribeTool] in the following ways.
Feedback. [ScribeTool] has feedback features so you can let us know what is working well and which features need improvement. If you use these feedback features, we may review your usage including the specific transcript, note or other Patient Information for the purposes of identifying or resolving the issue. A human will only review your Patient Information if you use these feedback features (such as thumbs up, thumbs down or written feedback). If you do not want a human to access Patient Information, do not use these feedback features.
Redaction. By default, some personally identifiable Patient Information is redacted from the text-based session transcript, enhancing privacy. However, some users may choose to turn off redaction. If you turn off redaction, this means the personally identifiable Patient Information will be stored on our secure servers until it is deleted. When changing redaction settings, consider the extra information that will be stored.”
I’ve said ‘confusingly’ above because the starting proposition is ‘we [use] anonymous data to improve our service’, but what follows are two instances that, contrary to the implication, do not seem to be exceptions to that starting proposition. We can see this by linking the opening proposition and the ‘However’ sentence with each of the bullet points:
- In the first case, the messaging is: ‘We use anonymous data to improve our service, but if you would prefer your data to not be used for these purposes, if you use our feedback features we may read identifiable patient information’. That doesn’t make much sense. Using feedback features and therefore exposing patient information to human review is clearly not an exception to ScribeCo ‘using anonymous data to improve its service’. Language like this raises more questions than it answers. For one thing, it leaves the reader wondering what the company means by ‘anonymous’ and why it thinks it’s OK for staff members to review potentially highly sensitive patient mental health information. I’ll return to that point further below.
- In the second case, the messaging is: ‘We use anonymous data to improve our service, but if you would prefer your data to not be used for these purposes, if you turn off redaction, the subset of personally identifiable Patient Information that would have been redacted will not be redacted and so will be stored on our services until it is deleted.’ Again, this makes little sense. Storing non-redacted personal information on ScribeCo’s servers has nothing to do with how ScribeCo might use anonymised data to improve it services. Again, the reader is left scratching their head.
Even putting this non-sensical language to one side for the moment, there is an obvious potential tension between saying in the FAQs “your data is not used to train AI”, without more, and saying in the fine print “we use anonymous data to improve our service.” What does “improve our service” even mean in this context? That’s not explained. It could mean evaluating output quality, refining prompts, tuning transcription pipelines, or any number of things. But the point is that “anonymous” patient data is being used, or so it would appear, for something beyond delivering the service to you, and you would not know that from reading the FAQ alone.
Now let’s return to the feedback feature. ScribeTool includes thumbs up and thumbs down buttons on generated notes, a standard piece of user interface that many people will click without a second thought. The privacy policy explains what happens when you use them: “If you use these feedback features, we may review your usage including the specific transcript, note or other Patient Information for the purposes of identifying or resolving the issue. A human will only review your Patient Information if you use these feedback features.”
Read that again. If you click a thumbs down on a note because the AI got something wrong, a person at ScribeCo may read the transcript of your patient’s therapy session. A person may review your “Patient Information”.* The privacy policy is clear about this. But how many clinicians, clicking a thumbs down icon to flag an inaccurate note, would expect that to trigger human review of their patient’s sensitive clinical data? How many patients would expect this to occur and what would they think about it if they knew?
The consent form that ScribeCo provides clinicians does not mention this potential for third party human review and so, when that consent form is used without more, patients are not told. This may not matter if the clinician never uses the feedback features, but it seems likely that at least some clinicians will not be aware of the detail here and could use the features, with neither they nor their patients being aware of the potential for third party human review.
Why this matters
This story is not about ScribeCo being a bad actor and I do not mean to imply that. From what I can see, they have built a product that takes privacy seriously in many respects: data stored on local infrastructure, encryption in transit and at rest, de-identification of transcripts before they are sent to third-party large language models (LLMs), and data processing agreements with those LLM providers. They provide consent forms and fact sheets for patients. Those are all good practices.
The problem is the gap between what a busy professional takes away from the FAQ and what the privacy policy actually permits (not to mention its lack of clarity). The FAQ creates an impression of simplicity: your data is not used. The privacy policy introduces defaults and exceptions that complicate that picture significantly.
This pattern is not unique to ScribeCo. It is quite common across the AI industry. Marketing pages and FAQs may give clean, reassuring answers. But terms of service and privacy policies contain the qualifications, carve-outs, and defaults that tell the real story. Most users will probably never read them.
Why New Zealand practitioners should care
Under the Privacy Act 2020 and Health Information Privacy Code 2020, a health practitioner who collects personal/health information must take reasonable steps to ensure that it is protected against loss, unauthorised access, use, modification, or disclosure (Information Privacy Principle 5). When a clinician uses an AI tool that processes session recordings, the clinician is responsible for understanding how that tool handles the data. The tool provider’s terms of service and privacy policy may be the primary source of that understanding.
If a practitioner relies on the FAQ and does not read the terms and privacy policy, they may not realise that anonymous patient data could be used to “improve the service,” or that clicking a feedback button could trigger human review of a therapy transcript. That gap in understanding could become a real problem if something goes wrong: a privacy complaint, a disciplinary matter, or simply an uncomfortable conversation with a patient who asks about how their data is being processed.
And then there’s the question of consent. If you’re relying on consent to record sessions and using feedback features that could result in third party human review of therapy transcripts, is the consent you obtain sufficiently ‘informed’ consent?
If I were a health practitioner using ScribeTool, I would never use the feedback features, I might be clear about that with patients, and if I managed a practice with multiple clinicians I would ask them to do likewise.
What to do about it
If you use any AI tool that handles sensitive information (whether patient health information or confidential business information), it can be important to take the time to read the privacy policy and terms of service. Not only the FAQ. Not only the marketing page. The actual legal documents. I appreciate that can be as enjoyable as sipping mud. If that’s the case, you can always enlist the aid of Claude, ChatGPT or Gemini. Personal review is desirable but if you can’t stomach it then using a decent LLM with a decent prompt on what to look for may be second-best (and in some cases may catch things you could miss).
Look for these things in particular:
- What happens to the data you enter and the output data beyond delivering the service to you?
- What commitments are given to you in relation to safeguarding that input and output data, keeping it confidential, and not using or disclosing it for other purposes?
- Are there features or processes (like feedback buttons, safety reviews, or quality improvement programmes) that change how your data is handled when you interact with them?
- Are options available that help you protect your input and output data, like turning off model training, or making chats temporary, or preventing staff from using feedback features, that you should be turning off or on?
- What security certifications or attestations does the AI tool provider have (and, separately) what is its security-related track record like?
These are not the only questions you need to ask when thinking of entering personal information into a third party AI tool, but they’re a good start. The answers might be perfectly acceptable. They might not. But either way, it can be important to know what they are before you press record.
(* “Patient Information” is defined in the privacy policy as “any information about a patient, whether that information is personally identifiable or de-identified. This may include personal health details, medical history, records of diagnosis and treatment, and other sensitive information that relates to the individual’s physical or mental health status.”)