In brief: An AI auditor’s independence is determined not by distance from the developer, but by the right to choose tests independently, obtain sufficient access, preserve original results, and report conclusions without agreement on a convenient wording. An embedded auditor may see more than an external one, but only if the contract separates funding from the decision, records conflicts of interest, prohibits swapping in a test build, and leaves an auditable trail.
This article is intended for executives, AI product owners, CISOs, risk/compliance teams, and procurement. It explains how to organize a model or agent review; it does not certify vendors, replace legal advice, or prove the safety of any specific system.
Contents
- What Anthropic and Accenture announced
- Why an external auditor may see too little
- Why employee-level access still does not mean independence
- Three levels of access to AI evaluation
- AUDIT: the contract for a verifiable evaluation
- How to review the agreement before work begins
- An independent evaluation plan for a company
- Evidence limitations
- FAQ
- How AgentSunrise helps prepare a system for evaluation
- Conclusion
What Anthropic and Accenture announced
On September 18, 2026, Anthropic announced a partnership with Accenture on embedded evaluation — an embedded independent evaluation of frontier models. The work will be led by Faculty, Accenture’s specialized AI division. The announcement includes red teaming, alignment assessments, and safeguards testing.
Anthropic and Accenture expect to invest at least $1 billion each over five years. Embedded evaluators are supposed to work inside AI companies with employee-like access: observe model training, development and release decisions, speak with teams, report incidents, and provide a public report on risks and benefits.
But the announcement itself openly acknowledges that the mechanism is still immature. There are no standards yet for access and reporting, no established funding model, and Accenture’s work is being paid for by Anthropic itself. The agreement is nonexclusive: both sides can work with other developers and evaluators. This is the start of an institutional experiment, not a ready-made seal of approval.
| What is known | What is not yet known |
|---|---|
| access is expected at the employee level | the exact set of data, systems, and model versions |
| red teaming, alignment, and safeguards are included | methodologies, sample sizes, and decision thresholds |
| the relationship is nonexclusive | who approves publication of a disputed result |
| funding initially comes from the company being evaluated | separation of budget, appointments, and final verdict |
| follow-up updates are promised | the first report, fixes, and independent replication |
The main practical takeaway: depth of access and independence are different properties. The first answers, “What can the auditor see?” The second answers, “Can the auditor call what they see a problem and preserve that conclusion in a conflict?”
Why an external auditor may see too little
A standard external review often gets a public API, a limited number of requests, and a vendor-provided description of the system. That is enough for some scenarios, but not enough if the risk depends on a hidden system prompt, moderation model, memory, tools, logs, weight version, or release process.
The UK guidance on frontier AI safety recommends matching access to the review task. The possible scope includes versions without safeguards, model families, internal metrics, the ability to fine-tune, and all components of the deployed system. At the same time, the document requires protecting weights and other information that could make dangerous misuse easier.
The result is not a binary choice of “give everything or nothing,” but a managed ladder of access:
- interface and documentation for checking observed behavior;
- configuration, system instructions, tools, and telemetry for product review;
- development and release artifacts for validating lifecycle claims;
- highly sensitive weights, data, or internal metrics only when there is a justified need and in an isolated environment.
High access creates its own risk: leakage of intellectual property, personal data, or dangerous capabilities. That is why it is granted not to “the auditor in general,” but for a specific question, timeframe, environment, and deletion protocol.
Why employee-level access still does not mean independence
An embedded evaluator may see a problem earlier and better understand the context. But closeness to the team creates pressure: dependence on budget, habits formed by internal assumptions, limits on publication, selection of a convenient model version, and so-called evaluator shopping — looking for the softest reviewer.
NIST AI RMF Core separates these issues. Measure 1.3 calls for involving internal specialists who were not part of frontline development and/or independent evaluators. Measure 2.1 requires documenting test sets, metrics, and tools, Measure 2.3 requires testing under conditions similar to real-world use, and Measure 2.13 requires evaluating the effectiveness of the TEVV process itself: testing, evaluation, verification and validation.
Anthropic’s own Advanced AI Framework proposes disclosure of funding and conflicts, cooling-off periods, pooled or government funding, and protection against evaluator shopping. These are useful proposals, but they come from the developer and do not yet prove that the new scheme meets them.
| Dependence risk | Minimum control | Verifiable evidence |
|---|---|---|
| the client chooses a convenient test | the test plan is fixed before launch | hash of the plan version and approval date |
| the developer swaps the test build | model and configuration identifier | signed manifest, call logs |
| an inconvenient result is softened | the auditor’s right to a separate opinion | contract clause and published position |
| payment affects the verdict | budget is separated from acceptance | governance structure, conflict disclosure |
| the fix remains only a promise | mandatory retesting | issue, evidence of fix, regression run |
| the audit becomes outdated after release | events that trigger an unscheduled rerun | version, date, change log, trigger policy |
The counterintuitive takeaway: sometimes an auditor inside the perimeter is more objective than a formally external one — if the former has an independent mandate and original data, while the latter sees only a prepared demonstration. Team geography does not replace the architecture of independence.
Three levels of access to AI evaluation
AI model audit — is an independent verification of claims about a system against predefined criteria, using reproducible evidence and a remediation path. It differs from ordinary testing because the developer cannot unilaterally choose the question, the evidence, or the outcome.
| Level | What the evaluator receives | What it is suitable for | What it does not prove |
|---|---|---|---|
| black-box | public interface, limits, documentation | user behavior, refusals, basic attacks | internal causes and full system scope |
| grey-box | system prompt, tools, versions, telemetry, test environment | product risk, agent flows, access control | training data quality and release decisions |
| embedded | lifecycle artifacts, teams, internal metrics and decisions | process checks, blind spots, and safety commitments | independence, if there are no separate rights and reporting lines |
No level is universally best. For a FAQ bot with no actions, black-box may be proportional to the risk. For an agent that changes a CRM, runs code, or affects a credit decision, you need at least grey-box scope and integration testing. Embedded access is justified when the audit concerns the training or release process itself.
AUDIT: a contract for verifiable evaluation
We propose the framework AUDIT. This is an original synthesis by AgentSunrise based on the Anthropic announcement, the NIST AI RMF, and UK materials; it is not an official standard from those sources.
A — adequate access
Tie each test question to the required artifact: interface, configuration, log, build, metric, or decision. Remove excess access and mark insufficient access as a limitation of the output.
U — independence conditions
Disclose the client, compensation, past and future commercial relationships, involvement in development, and the right to replace the evaluator. Team assignment and the final conclusion should not be controlled by a single business function.
D — test design
Before launch, record the hypotheses, sample, baseline, thresholds, stop conditions, and rules for handling exceptions. Save additional exploratory tests separately so a metric discovered after review is not presented as predeclared.
I — source artifacts
Preserve the model ID, configuration, tool versions, test set, raw outputs, human decisions, and hashes. A final PDF without the primary trace does not allow the conclusion to be reproduced.
T — interpretation and response
Separate observation, interpretation, severity, and management decision. Record the right to submit a separate opinion, any allowed disclosure limits, the remediation owner, and the trigger for re-evaluation.
Minimum task card:
| Field | What to record |
|---|---|
| claim | the specific assertion being tested |
| target | model, system prompt, tools, version, and date |
| evaluator | team, expertise, conflicts, and source of payment |
| access | artifacts and rights needed for each test |
| protocol | frozen tests, baseline, sample, threshold, stop rule |
| evidence | raw logs, hashes, human decisions, environment manifest |
| reporting | recipients, right to a separate opinion, redaction criteria |
| remediation | owner, deadline, rerun, and closure rule |
How to check a contract before work begins
We ran a deterministic demonstration on six synthetic cards. The check required at least grey-box access, separation of the evaluator from the developer, pre-frozen tests, access to raw logs, and the right to report the result.
Result: 6 out of 6 expected decisions matched. The full card was accepted; versions with black-box-only access, the developer participating in their own verdict, tests that could be changed after launch, no raw logs, or a ban on reporting were rejected.
This is not an audit, not a model benchmark, not a legal test, and not proof of independence. The demonstration shows only that key rights can be turned into machine-readable preconditions before work begins. The code and terms are stored in the article’s research artifact.
A practical contract acceptance test should include at least six scenarios:
- the auditor requests an additional artifact needed for the stated claim;
- the developer releases a new build during the test;
- an exploratory finding contradicts the main metric;
- the client disputes the severity, but not the fact of the observation;
- publication requires removing sensitive details without deleting the conclusion;
- the fix changes the system and triggers a regression evaluation.
If the contract does not answer who preserves the original observation in the event of disagreement, independence exists only in the presentation.
Independent evaluation plan for a company
Step 1. Start with a verifiable assertion
Do not order a “full AI audit.” Choose one decision: for example, the agent does not send an email without approval, or search does not return documents outside the user’s permissions. Link the claim to real harm and the risk owner.
Step 2. Separate three roles
The developer prepares the system and answers questions. The evaluator gathers evidence and states the conclusion. The risk owner accepts the residual risk and the go-live decision. One person can perform two roles in a small team, but the conflict must be visible and the decision recorded.
Step 3. Choose the access level
Provide exactly the artifacts without which the claim cannot be tested. For an agent, this usually includes tool configuration, a log of actual calls, identity/permissions, and the model version. The AgentSunrise article on managing AI agent autonomy shows why permission to act must be checked separately from answer quality.
Step 4. Freeze the protocol and environment
Version the tests, baseline, model, wrappers, and data. Protect the test environment from reaching live systems. After incidents in evaluation environments, it is especially important to distinguish simulation from external impact; the review order is described in the material on AI agent failures.
Step 5. Prepare the report before the result
Define the structure in advance: scope, exclusions, method, observations, uncertainty, conflicts, remediation, and separate opinion. Then a negative conclusion cannot be removed simply by changing the template.
Step 6. Repeat After the Fix
Closing an issue is not the same as fixing it. Re-run the original test and adjacent regression cases on the new version. For a high-risk system, schedule unscheduled review triggers for changes in the model, system prompt, tool, data source, or user class.
Evidence Limitations
- Anthropic’s new framework is described by Anthropic itself and has not yet produced an observable report, methodology, or result.
- The amounts promised are expected investments by the parties, not costs already incurred and not proof of the quality of the assessment.
- Direct funding creates a potential conflict; nothing in the announcement lets us conclude that it has been realized or already neutralized.
- The NIST AI RMF is a voluntary framework and is currently under review; it is not a certification.
- The UK guidance describes emerging practices, not a universally mandatory standard.
- The UK AISI case study tested four models and found no confirmed sabotage, but it also notes the limitations of the scenarios and evaluation awareness; the absence of a finding does not prove the absence of risk.
- The local 6/6 review assessed the structure of synthetic contracts, not the model, auditor, or production system.
- Russian practice, legal sufficiency, cost, duration, frequency of findings, and the impact of the audit on incidents were not observed.
Search volume, keyword difficulty, rankings, traffic, CTR, backlinks, and AI citations also Unknown. The article describes a verifiable procedure, not a promise of business or search results.
Frequently Asked Questions
What is an independent AI audit?
It is a review of claims about a model or system against predefined criteria, where the evaluator has sufficient access, preserves primary evidence, and can reach a conclusion independently of the developer.
Can the auditor be paid by the developer?
Yes, but that creates a conflict that must be disclosed and mitigated: separate the budget from the verdict, prohibit payment for a desired outcome, reserve the right to a separate opinion, and use pooled funding where possible.
How is red teaming different from an audit?
Red teaming looks for ways to break or provoke undesirable behavior. An audit checks specific claims and processes, includes criteria, an evidence trail, output limitations, and a remediation path. Red teaming can be part of an audit.
Is access to the model weights necessary?
Not always. For many product claims, grey-box access to configuration, tools, and logs is enough. Access to weights is justified only for a specific question and requires stronger safeguards.
Can a certificate be considered a guarantee of safety?
No. Any assessment is limited by the version, scope, methodology, time, and observable scenarios. After a significant change, a recheck is needed.
Where should a small company start?
Start with one critical claim and the five fields of the AUDIT framework: access, independence conditions, test design, source artifacts, and interpretation/response. Then run a limited grey-box review and a regression rerun.
How AgentSunrise helps prepare a system for evaluation
AgentSunrise can help turn an audit from a one-time demo into a verifiable control loop:
- describe one process, baseline, risk, and acceptance criterion;
- inventory the model, RAG, tools, data, roles, and versions;
- prepare the test environment, policy checks, logs, and regression set;
- integrate the remediation workflow, rechecks, and team training.
The first step is to choose one claim and fill out AUDIT before inviting the evaluator. Discuss the task.
Conclusion
Embedded evaluation solves a real problem: an outside observer often cannot see the system they are supposed to assess. But employee-level access does not create independence automatically. What is needed is a separate test plan, role separation, a primary evidence trail, the right to a separate conclusion, and mandatory retesting.
The working sequence is as follows: formulate the claim → choose proportionate access → disclose conflicts → freeze tests → preserve raw evidence → separate facts from decision → verify the fix. If the client can change any of these elements after the result, that is consulting with testing elements, but not an independent audit.