My AI chat has been under attack for two weeks: 8 rules for securing AI agents

For two weeks now, someone has been methodically trying to break the chat on my website. They try to make the model forget its instructions, reveal the internal prompt, carry out someone else’s command, or get to something the user should not see. From what I can tell, it has produced no useful result. Judging by the tone of the latest messages, the person is also upset 😞

The secret is not an “unbreakable prompt.” AI agent security starts with making sure that even a successfully deceived model still has no way to cause damage. A public chat has no shell, bash, or arbitrary code execution; tools are narrow; secrets never enter the context; dangerous actions are checked by code outside the LLM; the runtime is isolated; personal data is masked; and every step leaves a cleaned-up trace in the monitoring system.

In short: treat prompt injection as an expected event. Don’t try to prove the model will never make a mistake. Design the system so that a model error runs into permissions, policy checks, a sandbox, limits, and human approval.

What exactly they are trying to hack
Why a system prompt alone is not enough
8 AI agent security rules
How to choose between an LLM, an agent, and a code runner
How a request should flow
What to check in 30 minutes
FAQ
Bottom line

What exactly they are trying to hack

In a public AI chat, attackers usually do not target the model weights. They manipulate instructions and context:

they ask it to ignore the rules;
they disguise commands as a quote, code, a document, or data;
they try to pull out the system prompt, history, or a secret;
they push for a tool call with dangerous parameters;
they keep trying variations until one gets past the filter;
they try to write a harmful instruction into the memory of the next session.

OWASP defines prompt injection as manipulating LLM behavior through malicious input. The listed consequences include bypassing guardrails, data leakage, system prompt disclosure, actions through tools, and persistent influence across sessions.

It is important to separate two classes of problems.

Class	What happens	Main defense
The model was deceived	The LLM believed someone else’s instruction	guardrails, separation of instructions and data, output validation
The deceived model got privileges	The LLM called a dangerous tool, saw a secret, or went online	least privilege, tool broker, sandbox, approvals, egress policy

The first class cannot be considered fully closed. That is why the architecture must stop the second one.

Why a system prompt alone is not enough

A system prompt is useful as a role description, but it is not a security boundary. It is processed by the same model that reads untrusted text. If control exists only in natural language, the attacker is arguing with the model on its own turf.

OWASP AI Agent Security Cheat Sheet separately lists tool abuse, privilege escalation, data leakage, memory poisoning, goal hijacking, and excessive autonomy. This is no longer a matter of “writing a stricter prompt.”

The practical principle is this:

The LLM proposes intent. Trusted code decides whether it can be carried out.

The policy layer should receive the user’s original goal, the action type, parameters, identity, current limit, and risk. The decision allow, deny or require_approval is made by the main agent.

8 AI agent security rules

1. Do not give a public agent shell, bash, or arbitrary code execution

If the agent is accessible from the internet, a universal command interpreter turns a model error into operating system access. Even if you allow “just one command,” the shell often expands the attack surface through arguments, redirects, variables, substitutions, and child processes.

OWASP on excessive agency advises avoiding open-ended extensions such as shell command execution and replacing them with narrow functions. For a public chat, answering a question usually requires calling the LLM, not an agent with the right to execute code.

If code execution is truly part of the product, move it into a separate disposable job:

a new isolated environment for each run;
an empty file system or an approved set of files;
no secrets and no home directory;
network blocked by default;
strict CPU, memory, time, and output-size limits;
destroy the environment after the result.

2. Give only tools that are safe across all allowed parameters

The tool run_sql(query) is more dangerous than get_order_status(order_id). The tool http_request(url, method, body) is more dangerous than get_exchange_rate(currency_pair).

A good tool layer is built on the principle of least privilege:

one function for one business action;
a strict parameter schema;
a server-side allowlist of IDs and destinations;
read-only by default;
permission checks on every call;
an idempotency key for repeatable operations;
manual confirmation before deletion, payment, publication, or sending a message.

OWASP recommends limiting both the toolset and the functionality of each tool. If a chat does not need to delete emails, the delete function should not exist in the accessible model schema.

3. The agent runs in a sandbox or microVM — or is not used

A sandbox constrains a process inside a shared OS. A microVM provides a stronger boundary with a separate kernel and is better suited for untrusted code. The choice depends on risk, but the rule is the same: the agent should not run near production secrets and data just because that makes a prototype easier to build.

Task	Default mode
answer a question, classify, summarize	standard LLM call with no tools
read allowed data through a narrow API	agent with read-only tools and a policy broker
modify a record or send a message	narrow tool + permission check + confirmation
execute user code	one-time sandbox/microVM with no secrets and closed network

OpenAI describes sandbox as one layer that can detect unexpected communication and ask for user consent. But a sandbox does not replace network, permission, and time limits: an isolated environment with open egress can still send data out.

4. Guardrails and hooks are mandatory, but they serve different roles

Guardrail checks input, output, or a specific tool call. Hook observes the lifecycle: before and after an LLM call, a tool call, handoff, or agent completion.

Minimum set:

input guardrail — prompt injection, request type, rate limit, size, and encodings;
pre-tool guardrail — alignment with the original goal, permissions, parameters, budget;
post-tool guardrail — cleanup and validation of the tool result;
output guardrail — secrets, PII, dangerous markup, promises of completed actions;
hooks — logging, metrics, limits, audit, emergency stop.

OpenAI Agents SDK documentation shows input, output, and tool guardrails, and lifecycle hooks make it possible to observe the start and end of model and tool calls. At the same time, built-in shell and hosted tools may not pass through the same tool-guardrail pipeline. Check the actual execution path in your SDK, not just the presence of a nice-looking decorator in the code.

An LLM-based guardrail can also make mistakes. OWASP recommends using it as one layer, not a substitute for parameter checks, least privilege, and confirmation of irreversible actions.

5. The LLM must not see keys, tokens, or sensitive data

Do not put secrets:

in the system prompt;
in the conversation history;
in the tool description;
in variables available to the code runner;
in the full database output;
in traces without redaction.

The model should pass the intent: get order status 123. The trusted backend selects the credential, checks permissions, and calls the service. It is better to use a short-lived token with one scope for one run than a shared company API key.

A secret that is not in context cannot be coaxed out of context. That is stronger than any model promise to “never disclose tokens.”

6. Session compaction must preserve rules, not the attack payload

By “compaction” I mean a condensed summary of a long session that replaces older messages. This is useful for cost and context size, but it creates two risks:

an important constraint disappears after compression;
a harmful instruction from the dialogue gets into the summary and starts to look like trusted memory.

After each compaction, restore immutable security invariants from trusted code: available tools, permissions, prohibitions, original goal, limits, and the list of verified facts. Store untrusted text with provenance: who sent it, when, and with what level of trust.

Do not let the LLM itself decide which attack instruction to write into long-term memory. OWASP classifies memory poisoning as a separate risk in agentic systems.

7. The privacy filter goes before the LLM, and demasking goes after it

The working flow looks like this:

the detector finds a name, phone number, email, address, document, or other PII;
a local service replaces the value with a stable placeholder: [PERSON_1], [PHONE_1];
the LLM works only with masked text;
approved placeholders are restored locally before the response is returned;
the mapping does not go to the model provider and has a short TTL.

Microsoft Presidio supports replace, redact, hash, mask, and encrypt, and for reversible operations — deanonymization. Demasking requires a reversible operation; hashing and deletion will not restore the original value.

A privacy filter does not guarantee 100% detection. Add your own recognizers for Russian phone numbers, passports, tax IDs, contracts, and internal identifiers, and repeat the DLP check on output.

8. Observability should show the chain, but not become a new leak

For each request, it is useful to see:

session ID and trace ID;
prompt, model, and policy version;
the decision of each guardrail;
tool name, cleaned arguments, and result;
latency, tokens, cost, and number of steps;
approvals, denials, and the stop reason;
the compaction flag and the version of restored rules.

Langfuse links prompts, responses, tool calls, and retrieval steps into one trace, and traces into sessions. This helps reconstruct the causal chain, compare versions, and spot anomalies.

But do not send to the observability system what you are forbidden to send to the LLM. Before exporting traces, remove secrets and PII, restrict project access, set retention, and log views of sensitive events. For short-lived processes, do not forget to force-send buffered events before shutdown.

How to choose between an LLM, an agent, and a code runner

The safest optimization is not to introduce agency where it is not needed.

Question	If the answer is “no”	If the answer is “yes”
Does the system need to act, not just respond?	standard LLM call	move to the next question
Can the action be expressed as a narrow server-side function?	do not expose a universal tool	create a typed tool
Is the action reversible and low risk?	approval outside the LLM	limit + audit trail
Is untrusted code required?	do not allow code execution	a separate one-time microVM

NIST in the Generative AI Risk Profile considers security, privacy, evaluation, operations, and monitoring as part of the system lifecycle. That is a useful framework: protection does not end after model selection.

How a request should flow

Stage	What the system does	What is not trusted
1. Edge	rate limiting, WAF, request size, abuse score	IP and user text
2. Privacy	PII/secret detection and masking	raw values
3. Input policy	injection classification and allowed task	a decision from one LLM
4. LLM	forms a response or intent	its own assertion that “this is safe”
5. Tool broker	permissions, schema, scope, budget, approval	arguments from the model
6. Runtime	sandbox/microVM, closed egress, limits	the code and files being executed
7. Output policy	DLP, HTML sanitization, result validation	the model and tool output
8. Telemetry	redacted trace, alert, incident link	raw prompt and credentials

For a public chat, I would start with a zero budget for dangerous actions:

0 shell tools;
0 production secrets in context;
0 irreversible actions without confirmation;
maximum 5 tool calls per request;
maximum 30 seconds per agent run;
a separate attempt limit per IP, user, and session;
automatic blocking of a series of similar injection attempts.

This is a starting configuration, not a universal standard. Limits should be tuned to your normal traces and product scenario.

What to check in 30 minutes

Export a list of all tools available to the public agent.
Remove shell, code execution, arbitrary HTTP, and universal SQL.
For the remaining tools, document the allowed parameters and permissions.
Check which secrets appear in the prompt, tool output, and trace.
By default, close sandbox egress.
Add approval for deletion, payments, sending, and publishing.
Make sure rules are restored after compaction.
Put a privacy filter before the LLM and DLP after it.
Record one red-team scenario and trace it end to end in Langfuse or another tracing system.
Add a kill switch: revoke the token, disable the tool broker, and stop active runs.

FAQ

Can prompt injection be fully prevented?

You cannot promise that the model will never follow a harmful instruction. You can sharply limit the impact: remove dangerous tools, apply least privilege, verify actions in trusted code, isolate the runtime, and require human confirmation.

Is a sandbox enough for an AI agent?

No. A sandbox limits the process, but you also need closed egress, no secrets, resource limits, a disposable environment, and control over accessible files. Otherwise, an isolated process can still send data out or attack an accessible service.

Are guardrails and hooks the same thing?

No. Guardrails decide whether input, output, or a tool call is allowed. Hooks receive lifecycle events and are suited for logs, metrics, limits, and incident response. A policy system needs both mechanisms.

How Do You Keep Personal Data Out of an LLM?

Detect PII locally, replace values with placeholders, keep the mapping table outside the model, and restore only the permitted fields after the response. A second pass output check is necessary because the detector can miss a nonstandard identifier.

Should You Log Every Prompt in Langfuse?

You need to see the causal chain, but the raw prompt does not have to be stored in full. Mask PII and secrets before export, limit retention and access, and for investigations keep only the minimum amount of data needed.

What Should You Do About Security After Session Compaction?

Reapply security invariants from a trusted source and do not raise the trust level of user text just because it ended up in the summary. It is useful to record the compaction version and restored rules in the trace.

Takeaway

A person who spends two weeks trying to break my chat may eventually find a phrase that makes the model respond strangely. That is unpleasant, but it should not become an incident.

Real protection starts after the model: a public chat has no shell, no dangerous general-purpose tools, and no secrets; the tool broker checks every action; the sandbox restricts the environment; the privacy filter hides data; compactions do not erase rules; traces make it possible to reconstruct the chain.

The first step is to strip the public agent of every permission it does not need to keep the chat doing its job. Zero unnecessary capabilities gives you more security than one more paragraph in the system prompt.

8 Rules for Protecting AI Agents from Attacks