How to Control AI Costs as Token Usage Grows

The rise in AI usage almost always looks alarming on a financial chart: more employees, more agents, more requests, more context, more tokens. Many companies’ first reaction is to set hard limits, flood teams with alerts, and ask them to “be more careful.” That lowers the bill in the short term, but often breaks the main thing: the pace of AI adoption.

Short answer: stable AI spending is maintained not by banning usage, but by infrastructure. Set lower-cost models as the default, route requests by task, price, and cache hit, reuse warm context, keep sessions short, and make cost visible to teams. The goal is not fewer tokens, but fewer wasted tokens.

This article is a practical guide for CTOs, engineering leaders, Heads of AI, and operations leaders. We’ll break down why limits and alerts are a poor primary control mechanism, how to set up an LLM gateway, why cache hit rate matters, when you need a frontier model and when a cheaper model is enough, and which metrics to show engineers.

Contents

• Why AI spending grows faster than it seems
• Why limits and alerts don’t solve the problem
• Principle 1. Better defaults, not forced choices
• Principle 2. Route LLM requests by task
• Principle 3. Caching as the first layer of savings
• Principle 4. Minimal context instead of endless compaction
• Principle 5. Spend visibility and linkage to outcomes
• How to implement AI-cost governance in 30 days
• Which metrics to track
• Common mistakes
• FAQ

Why AI spending grows faster than it seems

AI costs rise not only because people send more requests. Usually several things happen at the same time:

• more employees get access to LLM tools;
• manual prompts turn into automated workflows;
• agents appear that make not one model call, but chains of calls;
• prompts accumulate instructions, examples, files, and chat history;
• teams start using frontier models for tasks where they are overkill;
• the same context is sent back to the model dozens or hundreds of times;
• debugging, reviews, tests, and retries consume tokens more quietly than production use cases.

As a result, the AI bill starts to look like cloud infrastructure before FinOps was implemented: every individual request seems reasonable, but the monthly total is a surprise. But with LLMs there is a difference. In classic cloud setups, you often optimize CPU, storage, and network. In LLM infrastructure, you optimize model choice, prompt shape, context reuse, history length, number of repeats, execution paths, and task quality.

AI-cost governance is a system of rules, routing, caching, observability, and product decisions that reduces the share of wasteful AI spending without suppressing useful use.

Here it’s important not to mix up two metrics:

Metric	What it means	How to think about it
Total tokens	Total number of input and output tokens	May grow if AI is delivering value
Wasted tokens	Tokens spent on overpowered models, repeated context, noisy retries, overly broad sessions	Should be reduced systematically

If a team is shipping more features, writing tests faster, analyzing documents faster, and serving customers better, token growth by itself is not a problem. The problem is when costs rise because of poor defaults and missing infrastructure.

Why limits and alerts don’t solve the problem

Limits and alerts are useful, but as a safety net, not as the main strategy. They answer the question, “When do we stop overspending?” but not the question, “Why are requests going through the expensive path in the first place?”

Imagine an engineering team where an engineer can choose any model. If the default in the UI is the most expensive frontier model, most people will leave it there. Not because they are wasteful. Simply because the default is the system’s hidden recommendation. Users perceive it as the normal choice.

Now add an alert: “You spent a lot of tokens.” What is the engineer supposed to do? Stop using AI? Manually compare models every time? Keep pricing, cache behavior, and task-specific quality in their head? That turns a developer into a model dispatcher, even though infrastructure should be doing that work.

Limits create another problem: they punish active users. The user-provided brief notes that 91% of employees never reached usage limits. If that’s the case, lowering limits won’t hit the bulk of spending, but rather the most active use cases, where AI may already be delivering value. It’s better not to choke adoption, but to make the cheaper route the default option.

Approach	What happens	Risk
Hard limits	The user hits a ceiling and waits to be unblocked	Adoption drops, workarounds appear
Frequent alerts	The team sees noise after the spend has already happened	Alert fatigue, few engineering changes
Cheap defaults	Most requests start with a more economical model	Escalation rules for stronger models are needed
Automatic routing	The system chooses a model based on task and cost	Observability and quality tests are needed
Cache-aware execution	Repeated context is reused	Prompt and architecture discipline are needed

A good policy sounds like this: “Engineers can choose any model if the task truly requires it. But the system should not send routine, repetitive, or simple tasks to the most expensive model just out of inertia.”

Principle 1. Better defaults, not forced choices

Defaults are one of the most underrated AI-cost levers. They work more gently than limits because they do not remove choice. At the same time, they change the behavior of most users.

The user-provided brief describes this approach: engineers can choose any model, but through the LLM gateway, cheaper open-weight models such as GLM 5.2 and Kimi 2.7 are tested as defaults. These models are not presented as universal replacements for frontier models. They become the starting point for tasks where their quality is sufficient.

This is an important distinction:

• a limit says, “you cannot use more”;
• a cheap default says, “start here, and if the task needs more, escalate”;
• routing says, “the user should not have to manually choose a model for every step at all.”

Where cheap defaults are usually a good fit

Scenario	What matters	Appropriate policy
Draft summarization	speed and low cost	economical model by default
Extracting structure from text	stable format	Cost-efficient model + JSON mode / validator
Ticket classification	Predictable categories	Cost-efficient model + selective validation
Text paraphrasing	Acceptable language quality	Cost-efficient model, escalate on request
Boilerplate code generation	Template-like output	Cost-efficient model + tests
Initial log analysis	Broad review	Cost-efficient model, then escalate for complex cases

Where you should not default to saving money

There are tasks where a frontier model is justified:

• High-uncertainty architecture planning;
• Complex code review;
• Security analysis;
• Tasks with a high cost of error;
• Final decision-making on legally or financially sensitive materials;
• Multi-step reasoning, where a weaker model will make more retries and may end up costing more.

For code review, model diversity is especially useful. If one model writes the code, another can review it. If every stage goes through the same model class, the same mistakes can repeat. Cost savings should not turn into monoculture.

Practical default policy

Start with a task matrix:

Task class	Default model tier	Escalation condition	Who can escalate
Summarization, classification, paraphrasing	Cost-efficient	Low confidence, poor formatting, user complaint	Automatically or by user
Draft coding	Medium	Tests fail, the task affects a critical module	Agent or engineer
Architecture planning	Strong	By default	Engineer / lead
Code review	Multiple models	High risk, security-sensitive diff	Pipeline
Final decision	Strong + human	Always human in the loop	Process owner

Main criterion: the default should be good enough for the mass-market class of tasks, but it should not block an informed choice of a stronger model.

Principle 2. Routing LLM requests by task

The next layer is routing. If defaults change the starting point, routing changes the selection mechanism itself. The user or agent describes the task, and the system decides where to send the request.

LLM routing is choosing the model, provider, and execution mode based on task type, quality requirements, cost, latency, availability, context window size, and cache hit probability.

The LiteLLM Router documentation describes an approach where the gateway can manage multiple deployments, fallbacks, and routing parameters. Link: LiteLLM Router documentation. The specific implementation may differ, but the principle is the same: model selection becomes an infrastructure function, not a manual responsibility for every engineer.

Why planning and execution should be separated

One of the most effective ways to reduce costs is to split the workflow into stages. For example, a frontier model may be needed for planning, but not for every small execution step.

Example:

1. A strong model analyzes the task and builds a plan.
2. A medium model handles routine transformations.
3. A low-cost model classifies, formats, and extracts fields.
4. A strong model is used only to check disputed fragments.
5. Validators, tests, and business rules filter out bad results without regenerating them with an expensive model.

This workflow is often better than "everything through the best model," because the expensive model is used only where it truly adds quality.

Example routing policy

Signal	Router decision
The prompt asks to "create a plan," "choose an architecture," "assess risks"	send to a strong reasoning model
The prompt asks to "extract fields," "convert to JSON," "group"	send to a cost-efficient model
The context matches warm cache	choose the provider/model with the best cache economics
The request contains security-sensitive code	use an approved model and enable enhanced logging
Need to respond to the user quickly	choose a low-latency route with fallback
The response did not pass validation	escalate to a stronger model or another prompt

A router should consider more than just the model price. Sometimes a more expensive model is cheaper overall if it solves the task on the first try, does not require retries, and does not break the downstream process. That is why you should calculate not the cost per one million tokens, but the cost of a successful outcome.

When people do not need to choose a model

Ideally, the engineer chooses the intent, not the model. For example:

• "fast and cheap";
• "high quality";
• "security check";
• "final answer to the customer";
• "draft";
• "experiment".

Then the LLM gateway maps the intent to the model, settings, cache, and fallbacks. This reduces cognitive load and makes the cost policy workable.

Principle 3. Caching as the first layer of savings

Cache misses are one of the easiest ways to drive up costs. If the system keeps sending the same instructions, the same files, the same examples, the same documentation, and the same history over and over again, it pays for repetition.

Prompt caching and prefix caching solve this problem: the repeated part of the context can be reused by the provider or runtime. The mechanics differ across platforms, but the operational meaning is the same: stable, repeatable prefixes and system instructions should be designed so they hit the cache.

Official resources:

• OpenAI Prompt Caching guide
• Anthropic Prompt Caching documentation
• vLLM Automatic Prefix Caching

The user-provided brief includes an example: in LibreChat, the cache hit rate rose from 5% to 60% after a correct implementation. This is not a universal benchmark for every system, but it shows how large the effect of fixing cache architecture can be.

What breaks caching

Caching often does not work not because the provider is bad, but because prompts are designed in an unstable way.

Problem	Why costs increase	How to fix it
Dynamic data is inserted at the beginning of the prompt	the prefix changes, so the cache cannot be reused	keep stable instructions at the top, data below
System instructions are regenerated each time	different wording breaks the match	version prompt templates
The entire history is added to the context	the prefix grows and changes	truncate history based on the task
Files are attached without filtering	the model reads extra material	retrieval based on relevant fragments
Different tools use different templates	the cache becomes fragmented	standardize gateway templates
No cache hit rate tracking	the team does not see the problem	add the metric to the dashboard

How to design cache-friendly prompts

A good prompt template can be thought of as layers:

1. A stable system instruction.
2. Stable formatting rules.
3. Stable examples, if they are truly needed.
4. A policy or workflow version.
5. Task-specific dynamic data.
6. The specific user question.

The point is for the repeated part to stay the same and sit where the runtime or provider can reuse it. If each application assembles prompts in its own way, the cache hit rate will be low even when there is a large volume of repeated tasks.

What to measure

Minimum set:

• cache hit rate by product;
• cache hit rate by workflow;
• cost of cached input vs uncached input;
• share of requests with a stable prefix;
• top prompts by uncached spend;
• cache misses after prompt template releases;
• the cost of retries due to invalid formatting.

Caching should be treated not as "some technical optimization down below," but as a product discipline. If the team changes a prompt template, it can affect the bill just as noticeably as changing the size of cloud instances.

Principle 4. Minimal context instead of endless compaction

Many teams try to fight costs through compaction: compress a long history and continue. This is useful, but it does not solve the root problem if too much unnecessary material gets into the session in the first place.

The right goal is not "use fewer tokens at any cost." The right goal is "use fewer tokens that do not help solve the current task."

What are wasted tokens

Wasted tokens are tokens that are paid for but do not improve the result: irrelevant files, old history, unused tools, repeated instructions, overly broad retrieval results, unnecessary reasoning, and retries after a poorly defined task.

Examples:

• the agent was given the entire repository instead of the three files it needed;
• the chat kept going after the task changed, even though a new session was needed;
• the system passes a list of every tool, including ones that are not used;
• a long policy is inserted into the prompt each time, even though it could have been cached;
• the model receives raw logs with 50,000 lines instead of an aggregated sample;
• the user asks to "check everything" when what is really needed is to check one diff.

Short-context practices

1. Start a new session when the task changes.
2. Limit file context to specific files and lines.
3. Disable tools that are not needed for the current workflow.
4. Use retrieval with hard limits and reranking.
5. Pass a summary only if it is needed for the next step.
6. Separate "project memory" from "current request context."
7. Log which documents actually influenced the answer.
8. Do not add large documents to every prompt "just in case."

Context Hygiene for Engineers

Instead of telling teams to "spend fewer tokens," give them simple rules:

• if the task is new, start a new session;
• if you are asking for a code change, attach only the relevant files;
• if you are asking for a review, provide the diff, the goal of the change, and the risks;
• if you are asking for research, separate facts from assumptions;
• if the model goes in the wrong direction, do not keep going forever—rephrase the task;
• if a large document is needed, first ask to extract the relevant sections.

These rules do not get in the way of work. They reduce noise, improve answer quality, and lower costs.

Principle 5. Cost Visibility and Connection to Outcomes

Cost transparency should not turn into public shaming of users. Its job is to give teams feedback: which workflows are expensive, where caching is not working, where the chosen model is excessive, and which scenarios deliver real impact.

Good visibility answers these questions:

• who is using AI;
• for which tasks;
• which models and providers are involved;
• how much a successful outcome costs;
• where cache misses happen;
• where there are lots of retries;
• which teams spend a lot but deliver measurable value;
• which teams spend a lot because of poor architecture.

The user-provided brief clearly states the right management position: engineers can use as many tokens as needed, from any model, but usage is visible; the more a team spends on AI, the greater the impact expected from it.

This is a mature approach. It does not say "do not spend." It says "show that spending turns into results."

What to show in the dashboard

Metric	Why it matters
Spend by team / product / workflow	identify the owner of the cost
Tokens by model	spot overreliance on expensive models
Cost per successful task	connect price to outcome
Cache hit rate	find the easiest savings lever
Retry rate	identify poor prompts and weak validators
Escalation rate	understand where the default model falls short
Latency by route	avoid optimizing cost at the expense of UX
Human override rate	see distrust in automatic routing
Quality incidents	do not cheapen the system at the expense of quality

Visibility alone does not reduce costs. It creates the foundation for engineering decisions: rewrite the prompt, enable caching, change the default, split the workflow, add a validator, or change the route.

How to implement AI cost governance in 30 days

Below is a practical plan without a heavy transformation.

Days 1-5. Inventory

Build a map of current usage:

• which products and teams call the LLM;
• which models are used;
• where there is manual model selection;
• which requests go through a shared gateway and which go directly;
• where logging is missing;
• which workflows generate the biggest spend;
• whether there is a cache hit rate;
• whether there is a retry rate;
• which tasks require frontier models.

The output of the week should not be a perfect dashboard, but a list of the 10-20 most expensive workflows.

Days 6-10. Task classification

Break workflows into classes:

• routine transformations;
• data extraction;
• classification;
• summarization;
• draft generation;
• coding;
• code review;
• planning;
• security-sensitive analysis;
• final customer responses.

For each class, define the default tier, the escalation condition, and the minimum quality criteria. This is the foundation of routing.

Days 11-15. Defaults and gateway

Move standard tasks to cheaper defaults. If you already have an LLM gateway, add rules. If not, start at least with a single SDK wrapper or proxy layer that logs the model, cost, route, workflow, and user/team.

Important: do not remove the ability to choose a stronger model. Just make it a deliberate choice, not a hidden default.

Days 16-20. Caching

Choose 3-5 workflows with repeatable context:

• an enterprise chat with the same system instructions;
• a code assistant with recurring repo instructions;
• customer support with the same rule base;
• analytical agents with repeatable schema descriptions;
• LibreChat or a similar interface with shared prompts.

Check:

• whether prefixes are stable;
• where dynamic data is inserted;
• whether prompt templates are versioned;
• whether there is a cache hit rate metric;
• whether the cache changes after a release.

Days 21-25. Context hygiene

Set short rules:

• new task - new session;
• files are attached narrowly;
• tools are enabled by workflow;
• retrieval is limited;
• large documents are first broken into relevant parts;
• a summary does not replace a clean start if the task has changed.

Add prompts in the UI and templates, but do not turn it into a lecture. The best rule is the one built into the interface.

Days 26-30. Visibility and review

Show the dashboard to teams. Do not start with blame. Start with questions:

• which workflows cost the most;
• where there is a lot of uncached input;
• where expensive models are used for simple tasks;
• where there are many retries;
• where savings may hurt quality;
• where the costs are justified by the impact.

Then choose 3-5 changes for next month. AI cost governance is best rolled out iteratively: one defaults policy, one routing rule, one cache fix, one dashboard improvement.

Which metrics to track

For sustainable AI growth, it is not enough to look only at the total bill. The total bill answers the question "how much did we spend," but it does not show what to do.

Metric	Good question
Total AI spend	is cost growing faster than useful usage?
Spend per workflow	which scenarios make the biggest contribution?
Cost per successful task	how much does a completed result cost?
Input/output token ratio	are we bloating the input context?
Cache hit rate	are we reusing repeatable context?
Uncached repeated prompt spend	how much are we paying for something that could have been cached?
Retry rate	how much do we spend fixing bad answers?
Model escalation rate	where are defaults too weak?
Frontier model share	where are strong models being used out of habit?
Quality incident rate	did the system get worse after optimization?

Track the "cost of outcome," not just the "cost of the request." A cheap model that breaks the format and requires five retries can end up costing more. A more expensive model that once correctly plans the work for ten cheaper steps may be cost-effective.

Common mistakes

Mistake 1. Thinking the goal is fewer tokens

If AI is delivering value, token usage will grow. The goal is not to stop growth, but to make it sustainable. The waste to cut is extra context, the wrong models, repeated prompts, retries, and lack of cache.

Mistake 2. Making the expensive model the default for everything

The frontier model may seem like the safe choice, but at scale it becomes an expensive habit. Reserve it for the tasks where it is truly needed.

Mistake 3. Giving users too many manual decisions

If every engineer has to know the pricing of every model, cache behavior, and selection rules, the system will not scale. Automate routing.

Mistake 4. Implementing cache without prompt template discipline

Cache will not help if every request starts with a unique, randomly assembled prefix. You need stable instructions and template versioning.

Mistake 5. Continuing old sessions after the task changes

A long history creates the illusion of memory, but often adds noise. If the task context has changed, a clean start is better than compacting it.

Mistake 6. Cutting corners on review and security

Cost optimization should not reduce reliability. For code review, security, and high-impact decisions, use stronger models, model diversity, and human oversight.

Mistake 7. Comparing only token price

Compare the cost of a successful result. That includes retries, latency, quality, manual fixes, and downstream errors.

Practical architecture

The minimum architecture for sustainable AI cost looks like this:

1. A single LLM gateway or SDK wrapper.
2. A task catalog and default routes.
3. A rules engine for routing.
4. A provider/model registry with price, latency, context window, cache support.
5. A prompt template registry with versions.
6. A cache-aware request builder.
7. Validators for format and quality.
8. Observability: tokens, cost, cache, retries, route, team, workflow.
9. A dashboard for teams and leaders.
10. A review process for changing defaults and routing rules.

This layer can be rolled out gradually. You do not have to build a complex platform right away. But if direct calls to different LLM providers spread across dozens of services, managing cost will become almost impossible.

What AI Dawn Can Do

AI Dawn helps companies design and implement AI infrastructure that reduces costs without blocking usage:

• audit current LLM spend and workflows;
• design an LLM gateway;
• task-based routing rules;
• implement prompt caching and cache-aware templates;
• set up dashboards for AI cost governance;
• develop internal AI assistants and agents;
• securely integrate LLMs into development, sales, support, and document workflows.

If your team is already seeing rising AI costs but does not want to slow adoption, the best place to start is not with usage limits, but with a workflow map: where tokens are being spent, which tasks really require stronger models, where the cache is not working, and where the context is bloated.

FAQ

How can you reduce AI costs without banning usage?

Reduce wasted tokens, not AI usage. Make cheaper models the default, route requests by task, turn on caching, keep context short, and show teams the cost of their workflows.

Why are usage limits not the main tool?

Limits are useful as an emergency brake, but they do not improve the system very much. They do not fix expensive defaults, lack of routing, cache misses, or overly broad context. That is why limits should be used together with infrastructure changes, not instead of them.

What matters more: defaults, routing, or caching?

You need all three layers. Defaults lower the baseline cost of high-volume scenarios. Routing chooses the right model for a specific step. Caching reduces the repeat cost of the same or similar context. In practice, the biggest impact comes when these layers work together.

When should you use a frontier model?

A frontier model makes sense for planning, complex reasoning, architectural decisions, security-sensitive analysis, important code reviews, and tasks where mistakes are costly. For field extraction, formatting, classification, and simple summaries, it is often overkill.

What is cache hit rate in LLMs?

Cache hit rate shows what share of requests were able to reuse a previously processed part of the context. The higher this metric is in repeatable workflows, the less the team pays for the same instructions, documents, and prefixes.

Why not just compact long sessions?

Compacting is useful, but it does not replace a clean start. If the task has changed, old history often adds noise and burns tokens. It is better to start a new session, limit files, and pass only the relevant context.

How do you know savings did not hurt quality?

Track not only spend, but also retry rate, escalation rate, human override rate, quality incidents, and cost per successful task. If costs went down but errors and manual fixes went up, the optimization may be false economy.

Can engineers be allowed to use any model and still keep costs under control?

Yes, if the choice is visible and the infrastructure helps route requests correctly. An engineer can escalate a task to a stronger model, but high-volume scenarios should start with an economical default, and routing and caching should work automatically.

Conclusion

Exponential growth in token usage does not have to mean exponential growth in costs. But to make that happen, AI cannot be managed with limits and alerts alone. It needs infrastructure that makes the right behavior the easiest behavior: low-cost defaults, automatic routing, caching, short context, and a transparent link between spend and outcomes.

The user-provided brief says that implementing this approach cut AI costs by almost half, even as token usage continued to grow. Even if the exact result differs from company to company, the management takeaway remains the same: the goal is not to suppress AI usage, but to build a system where growth is sustainable.