Briefly: checking each AI agent skill separately does not prove the safety of the end-to-end workflow. Risk emerges when one skill reads sensitive data, a second turns it into a normal work artifact, and a third sends, executes, or stores the result. That’s why before launch you need to check not only packages, but also the allowed data and effect chains in runtime.
This material is intended for CTOs, CISOs, process owners, and teams connecting skills, plugins, MCP servers, and business systems to agents. It explains architectural validation of chains; product-specific auditing, legal assessment, and promises of complete protection are out of scope.
Contents
- What the CompoSkill study showed
- Why safe skills do not make a safe system
- The source—bridge—effect model
- Which chains to look for in a business process
- The CHAIN method for validating the route
- Which runtime controls to put in place
- Why approval and scanning are not enough
- How to run a safe pilot
- What to measure
- Study limitations
- Frequently asked questions
- How AI Dawn helps validate an agent workflow
- Conclusion
What the CompoSkill study showed
On August 17, 2026, a preprint was published on arXiv CompoSkill: Compositional Skill Chain Attacks from Individually Scanner-Passing LLM Agent Skills. Its main thesis: the status of “passed the scanner” applies to one skill, but risk can appear only after several skills are combined into a route.
The authors assembled CompoSkill-Bench with 1,140 records: 380 professional tasks in clean, explicit, and implicit attack variants. The set covers five threat types, six work scenarios and 76 roles across two agent environments—OpenClaw and Nanobot. In experiments, the rate of forming a risky chain reached 83.3% in white-box and 80.6% in black-box mode. These are metrics for a specific benchmark, not an estimate of attack probability in a company.
The black-box scenario is especially important: the attacker did not name skill identifiers. They phrased a plausible work task, and the agent itself found the right combination. In the DevOps example, one skill read configuration, a second formatted it as an audit report, and a third sent the report outside. Each component could look normal on its own.
The study compared three scanners in strict blocking mode. After the flagged packages were removed, the chain formation rate in one configuration fell from 63.2% to 36.8–40.8% for two scanners and to 31.6% for the third. The authors interpret this as preserving 50–65% of the attack's compositional capability. The point is not that scanners are useless: they reduce the risk surface, but they do not observe the full path the agent will build during execution.
Why safe skills do not make a safe system
Compositional skill risk — a dangerous data or permission flow that appears only when several allowed components are called in sequence. One skill may have the right to read a file, another to transform data, and a third to send a message. None of them combines reading a secret with sending it outside, but their route does.
This is similar to transitive permission that no one explicitly granted:
read config
→ compile report
→ send to contractor
local permissions: allowed
end-to-end effect: the secret left the trusted zoneThat is why a package inventory answers only the question, “what is installed?” For security, three more questions are needed:
- What states can each component read?
- What can its output turn into at the next step?
- What external, persistent, or privileged effect completes the route?
The same principle is already visible in official guidance. Google Cloud separately calls insecure tool chaining a risk in agent-only mode and recommends a dedicated identity with least privilege. The MCP specification prohibits token passthrough and requires binding the token to the target resource so a confused deputy is not created. NIST NCCoE centers identity, authorization, audit, and nonrepudiation of agent actions. These are independent confirmations of the architectural direction, not confirmation of the CompoSkill percentages.
The source—bridge—effect model
For an initial check, it is useful to break the route into three roles.
| Role | What it does | Typical capabilities | Main question |
|---|---|---|---|
| Source | receives sensitive or control state | files, config, database, memory, email | what data can leave the trusted zone? |
| Bridge | normalizes, summarizes, archives, or turns data into a command | report, script, spreadsheet, attachment, prompt | does the transformation hide the risk of the original state? |
| Effect | creates an external, persistent, or privileged result | network, shell, message, memory write, permission change | can the action be stopped or rolled back? |
The bridge is especially dangerous not because it is malicious. It makes the transition feel natural. In CompoSkill, the rate of building a leakage chain rose from 64.9% for two skills to 80.6% for three; for privilege escalation — from 65.8% to 71.1%. In the extended experiment, the average attack success rate rose from 35.2% at two steps to 55.4% at three, then declined to 37.8%, 33.6%, and 21.8% at four, five, and six steps. The authors call this bridge-bonus-then-hop-decay.
Practical interpretation: protect not every long sequence, but first of all short natural routes where the transformer connects the source to a strong effect. However, the study’s numerical thresholds should not be turned into a universal rule that only three-step paths are dangerous.
Which chains to look for in a business process
| Threat | Source | Bridge | Effect | Example business invariant |
|---|---|---|---|---|
| Data leakage | CRM, email, secrets, contracts | summary or export | external email, webhook, cloud drive | confidential fields do not leave the approved destination |
| Memory tampering | policy or agent memory | a “helpful” instruction | long-term record | external data do not change the policy without a separate owner |
| Privilege escalation | roles and environment | script generator | shell, IAM, admin API | the granted permission is no broader than what the user delegated |
| Takeover of another agent | handoff and routing state | shared task | inter-agent message | the recipient rechecks the source and scope |
| Resource exhaustion | quota and task status | loop or retry plan | repeat calls | budget, depth, and number of retries are limited by policy |
Do not stop at tool names. report-builder sounds harmless, but its output can save secrets in a document. email-sender does not know the attachment is sensitive. So it is more useful to describe capabilities as a flow: read:secret → transform:document → send:external.
For each path, record the trusted zones. Sending from CRM to an internal DLP boundary and sending the same object to any arbitrary address are different effects. Writing to a temporary scratchpad and changing an agent’s long-term memory are different levels of persistence.
The CHAIN method for route validation
We propose the CHAIN. It is an editorial synthesis of recent research and official security guidance, not an industry standard.
C — Goal
Write down the original business goal and the acceptable final result. The wording “prepare an audit” should not automatically include external sending or reading every configuration.
H — Natural route
Build the normal execution path at the capability level: what is read, transformed, stored, and sent. Separately note the steps the agent may add on its own.
A — Authorizations
For each call, specify the identity, resource, scope, duration, and delegating party. Issue a token to the target service; do not pass a universal user token along the chain.
I — Interruption
Define a control before the strong effect: policy denial, human approval, two-person rule, destination allowlist, or a safe fallback. Checking after the send detects an incident, but does not prevent it.
N — Sensitive data
Assign labels to inputs and intermediate artifacts. Summarization, archiving, or format conversion should not remove the classification of the original data.
K — Kill channel
List terminal capabilities: external network, messages, shell, database write, IAM change, publishing, memory. The worse the reversibility, the narrower the interface should be.
A — Audit
Preserve the actual trajectory: actor, delegated user, policy version, input labels, calls, arguments, control decision, recipient, result, and rollback handle. Do not copy secrets into the log; a secure link, digest, or redacted evidence is enough.
What controls to put in runtime
A batch scanner remains the first layer. It is useful for detecting dangerous code, excessive privilege declarations, known signatures, and suspicious instructions. But the decision on a specific route should be made closer to the effect.
User request / external data
↓
Planner
↓
Policy engine across the full trajectory
↙ deny ↓ allow ↘ approval
narrow tool adapters with separate identities
↓
Target system rechecks scope
↓
Postcondition verifier + audit eventMinimum runtime stack:
- Separate identities. The agent and the user do not share a perpetual super-token.
- Narrow tools.
send_approved_report(destination_id, artifact_id)is safer than a universal SMTP client. - Data provenance labels. Data classification is carried through transformations.
- Destination verification. The allowlist takes into account the domain, tenant, project, and recipient type.
- Prohibit dangerous combinations. The policy evaluates the recent data source and the requested effect together.
- Budget limits. Limits on hops, retries, cost, time, and the volume of external operations.
- Independent verification. The result is read from the target system, not from the agent's text.
OWASP AI Agent Security Cheat Sheet recommends repeating security testing after significant changes to prompts, tools, memory, retrieval, policy, or model provider, and separately checking tool misuse, privilege escalation, memory poisoning, data exfiltration, and recursive tool abuse. This means the allowed graph cannot be certified once and for all.
Why approval and a scanner are not enough
Human approval is useful when a person can see a meaningful representation of the effect. The request 'allow email-dispatcher?' tells you almost nothing. You need the recipient, attachment classification, data provenance, diff, approval basis, scope of permission, and the ability to decline.
Bad approval becomes a rubber stamp if:
- there are too many requests;
- the agent shows only the tool name;
- sensitive fields are hidden inside an archive or summary;
- the confirmation applies to the plan, not the exact action;
- one approval opens a series of future calls.
The scanner and approval solve different problems. The scanner evaluates a component before installation; approval checks a single action; the policy engine constrains the route; the target system protects its own invariants. Removing any layer increases dependence on the others.
The same principle applies to prompt injection. An injection can deliver a malicious goal, but damage only occurs if the infrastructure allows data and effect to be connected. Therefore general AI agent protection measures must be supplemented with path-level controls. And graph engineering helps make dependencies and execution points explicit.
How to run a safe pilot
- Choose one process. Record the owner, baseline, data, constraints, and acceptance criteria.
- Create a capability inventory. For each skill, describe read, transform, write, send, execute, and persist.
- Build a route graph. Start with short paths source → effect and source → bridge → effect.
- Document prohibited combinations. For example:
customer_pii → external_messagewithout DLP and separate approval. - Grant minimum privileges. Create agent identity, resource-bound tokens, allowlists, and TTL.
- Run replay. Run normal tasks and five classes of attack scenarios on data copies.
- Switch to shadow mode. The agent builds the route, but strong effects are executed by a human or a simulator.
- Open limited execution. Allow only observably safe paths with a kill switch and rollback.
- Retest changes. A new skill, policy version, model, or integration creates a new graph.
The rollout order can be linked to a business-process audit before AI: first process boundaries and baseline, then tools and autonomy.
What to measure
| Metric | How to calculate | What it shows | What it does not prove |
|---|---|---|---|
| Forbidden path rate | forbidden routes / all proposed routes | the planner's tendency to assemble a risky path | the likelihood of a real incident |
| Pre-effect block rate | blocked before effect / all forbidden routes | runtime policy performance | absence of bypass channels |
| Approval precision | meaningful rejections / all approval requests | quality of routing to a human | safety of automatically approved paths |
| Utility completion | correctly completed clean tasks / clean tasks | the control cost for a useful process | the economic impact of implementation |
| Evidence completeness | events with actor, scope, inputs, and result / all effects | investigability | correctness of the agent's decision |
| Recovery success | successfully rolled back tests / rollback tests | recovery effectiveness | reversibility of external communications |
Thresholds should be set based on process criticality and your own baseline. You should not use 83.3% or other CompoSkill figures as the target KPI: those are characteristics of an experimental setup.
Research Limitations
CompoSkill is published as arXiv v1; as of the observation date, independent peer review has not been confirmed. The experiments are limited to two agent environments, selected models, marketplace skills, synthetically assembled attack variants, and study judges. They demonstrate the existence and reproducibility of a risk class, but do not measure how common successful attacks are in real companies.
ASR and CFR measure different things: an agent may assemble a risky chain but never carry it through to a harmful effect. Lower numbers for one model in the table do not prove universal safety. Conversely, a high value in one scenario does not predict the outcome in your configuration.
Official guidance from Google Cloud, MCP, NIST, and OWASP confirms the need for least privilege, token-to-resource binding, auditing, and chain verification, but does not validate the CompoSkill methodology or percentages. Before moving to production, you need your own replay testing, adversarial tests, and validation of the target systems.
Frequently Asked Questions
What Is an AI Agent Skill
A skill is a pluggable instruction, tool, or capability package that helps an agent perform a class of tasks. Risk is determined not only by the skill code, but also by access to data, side effects, and connections to other skills.
Is It Enough to Scan Each Skill
No. A scanner reduces the risk of an individual component, but may miss a route that appears only during execution. You need a capability graph, runtime policy, and control over strong effects.
How Is a Compositional Attack Different from Prompt Injection
Prompt injection delivers someone else’s instruction to the agent. Compositional risk describes the infrastructure path by which several permissible skills turn an instruction into harm. Injection is possible without a successful effect if the route is constrained.
Which Chains Should Be Tested First
Short paths from secrets, configurations, databases, or memory to the external network, shell, messages, long-term storage, and permission changes. Special attention should go to the transformer between the source and the effect.
Should You Block the Agent from All External Actions
Not necessarily. It is safer to give a narrow interface, a separate identity, an allowlist of recipients, TTL, rate limits, data classification checks, and approval for exceptions than to provide a universal tool with broad permissions.
When Should You Repeat a Security Test
After changing the skill, model, prompt, memory, retrieval, policy, permission set, MCP server, or business integration. Any such change can create a new permissible path.
How Does AI Dawn Help Validate an Agent Route
AI Dawn can build path-level control into a specific business process:
- Audit the process, data, baseline, constraints, and acceptance criteria.
- Map the capability graph of skills, MCP servers, and integrations with business systems.
- Design narrow tool adapters, agent identities, policy checks, approvals, logging, and recovery.
- Run the MVP, testing, launch, team training, and change support.
A safe first step is to choose one process, its current baseline, data sources, constraints, and acceptance criteria. Then you can build a graph of short routes and test them on a data copy before enabling real effects.
Conclusion
Skill safety is not closed under composition: two or three permissible components can create a capability that none of them has individually. That is why the real unit of control is not a package in a catalog, but the trajectory from the data source to the terminal effect.
The practical sequence is: inventory capabilities, map short paths, preserve data classification through transformations, grant separate minimum permissions, place deterministic controls before the effect, and record the actual trajectory. A scanner remains a useful layer, but the production decision should rely on replay and runtime observation of your configuration.