Introduction
Most agencies understand that a proposal to process personal information with an AI tool calls for a privacy impact assessment (PIA). It’s equally important to remember that undertaking a PIA and implementing agreed-upon recommendations is not necessarily the end of the matter. A PIA describes a tool as it was on the day it was assessed but, more so than other technologies, AI tools do not stand still or evolve slowly. New models are released, terms change, and data handling that was true last month may not be true today. A PIA that was sound when written can quietly become out of date, and an agency that keeps relying on it may be carrying privacy risk it has not assessed.
We’re going to use Microsoft Copilot as an example, focusing on the recent inclusion of Anthropic’s (currently unavailable) Fable 5 models, and explain why it matters.
The usual position: the AI model provider as licensor or subprocessor
When a New Zealand agency uses an AI tool like Microsoft 365 Copilot, it normally relies at least in part on the platform provider’s contractual commitments. In the case of Copilot, the agency contracts with Microsoft, and usually the underlying model providers sit behind Microsoft, either as licensors of locally hosted versions of their models or as subprocessors. Microsoft’s data protection commitments, such as restrictions on how customer content can be used and disclosed, usually apply across the board.
A PIA of a Copilot deployment might assume this structure and assess risk on that basis. However, whilst that assumption is reasonable for most models, it is not reasonable for all of them.
Where the AI model provider is not a subprocessor
Whilst Anthropic is treated by Microsoft as a subprocessor in relation to most of its models, it is not treated as a subprocessor for the models that Microsoft labels as “Preview models with Data Retention”. For those models, Anthropic acts as an independent processor, and the model-provider processing is governed by Anthropic’s commercial terms and data processing agreement rather than the ordinary Microsoft subprocessor arrangements.. This is a recent change. It appears to arise from Anthropic’s release of its Fable 5 models, which Anthropic designates as “Covered Models”, and it will apply to any future models Anthropic designates the same way. (We can put to one side for the moment the fact that Fable models have been disabled for customers worldwide due to US government export control restrictions. It seems likely they’ll be back on deck, in some shape or form, before too long.)
The distinction is visible to the organisation’s Copilot administrator at the point of model enablement. Microsoft’s admin interface presents Anthropic Models with Data Retention as a separate, optional preview-model category, requires acceptance of additional terms, and states that the relevant Microsoft customer agreement (including the Product Terms and Microsoft Data Protection Addendum) do not apply to use of those models within Microsoft Online Services. It also states that use is subject to Anthropic’s Commercial Terms of Service, Data Protection Addendum and data-retention policy, and that Anthropic does not guarantee Zero Data Retention.
Implications
This places significant responsibility on an organisation’s Copilot administrators. The Microsoft admin interface gives clear notice of the different contractual and data-handling position, but the decision to enable these models is still an administrative action that could have legal, privacy, security, records management, data residency, and assurance implications for the organisation as a whole.
Organisations may therefore need to update their AI policies, Copilot governance settings, and/or internal approval processes. In particular, Copilot administrators should be instructed not to accept additional model-provider terms, or enable models involving provider-side data retention, unless there has first been appropriate legal, privacy, security and governance review. This is especially important where the relevant model is surfaced within the same Copilot environment as other models, because end users may experience it as just another Copilot model choice and may not appreciate that a different contractual chain and retention model applies.
On the Anthropic side, Anthropic’s retention and review rights for these new models come from its Service Specific Terms (clause F, “Covered Models”). Under that clause, the customer agrees that Anthropic may retain and conduct safety reviews on inputs, outputs, and related data. The clause expressly overrides any zero-data-retention commitment that would otherwise apply to a customer that had negotiated one. So a zero-retention arrangement an agency thought it had may simply not hold for these models.
In practical terms:
- Anthropic, not Microsoft, retains inputs and outputs for these models for up to 30 days before automatic deletion.
- Content flagged for potential serious harm may be subject to human review by a limited set of approved Anthropic reviewers. The tooling blocks export or copying and records each access in a tamper-proof log, with automatic deletion after 30 days, except where the data forms part of a safety investigation or Anthropic is legally required to keep it.
- The models are disabled by default at the Copilot tenant level. They require an explicit, separate administrator opt-in before any user can access them.
None of that is hidden, and Anthropic’s controls around the review process are careful. But it is materially different from the basis on which many recent Copilot PIAs will have proceeded.
Why this is a PIA issue, not merely an administrator issue
An agency completes a PIA on its Copilot deployment. The PIA assesses risk on the basis that model providers sit behind Microsoft as licensors or subprocessors, with Microsoft’s commitments applying to the agency’s use of the models within Copilot. The PIA and its recommendations are approved. Months later, a new class of models becomes available in the same environment, sitting outside that contractual structure, with different governing terms, including as to retention and human review. An administrator, seeing a new capability appear, opts in.
At that point the agency may be processing personal information under contractual and data-residency arrangements its PIA did not consider. The PIA still exists, and on its face it covers “Copilot”. But it does not actually cover this change. Relying on it may give a false sense of comfort.
In this kind of situation, enabling the ‘Covered Models’ should be considered a distinct processing activity. It involves a different processor operating under different contractual terms, different retention, and a possibility of human review of content that the original assessment did not contemplate. A change of that kind should be assessed on its own terms, rather than being assumed to be covered by an existing PIA.
Practical recommendations
For the agencies I advise, my recommendations in this situation would be twofold:
- First, in relation to the agency’s use of Copilot, do not opt in to Anthropic’s “Covered Models” (such as Fable) on the strength of an existing Copilot PIA. Treat enablement of those models as its own processing activity that needs its own privacy impact assessment or update to an existing assessment, taking account of the different contractual terms, additional offshore retention of inputs and outputs, and the potential for limited, controlled human review outside Microsoft’s contractual commitments. Whether the residual risk is acceptable is a decision for the agency to make with its eyes open, and for some agencies and some use cases it may well be. The point is to make that decision deliberately rather than by default. If additional assessment is not undertaken, and something goes wrong later, the agency would likely find it difficult to maintain that its original PIA remained current and complete.
- Second, and more generally, build a trigger for review into how the agency manages its AI tool usage. For many this is obvious, but a PIA should not be a document that is completed once and filed. When a provider releases a new model under different terms, changes its terms, or introduces a new data handling regime, that is a prompt to ask whether your existing assessment still holds. The “data retention preview” label is exactly the kind of signal that should prompt a fresh look, and a general question along the lines of “has anything changed” could be included as part of periodic governance reporting and review.
The wider point
The Fable models example is useful because it’s concrete, but the lesson is not really about those particular models. It is about the nature of AI tools. The thing you assessed and the thing your staff are using can drift apart without anyone deciding that it should. New models arrive inside familiar products. Terms shift underneath you. The contractual chain you relied on for one feature may not apply to the next.
A PIA captures a moment in time. Keeping it useful means revisiting it when the tool materially changes, and treating a genuinely new processing activity as exactly that. For agencies handling personal information, that discipline is what keeps a PIA connected to the real system in use, rather than the system that was assessed when the PIA was written.
This article is general information, not legal advice. If your agency is weighing up whether to enable particular AI models or needs help keeping its privacy impact assessments current, I’m happy to talk.