Having your own LLM in a company is not just a model on a server, but the service built around it: separate GPU pools for different workload types, a scheduler with a queue and deferred tasks, a gateway with permissions and quotas, request routers, and mandatory quality checks after every change. That is the conclusion Sergey Notevsky, AI Platform Lead at Bitrix24, draws in his October 12, 2026 analysis. His main business takeaway: moving from an external API to your own infrastructure does not automatically make tokens cheaper—queues, caching, and handling spikes become your responsibility and part of your cost structure.
Prepared by AIrassvet October 13, 2026 based on a Bitrix24 post on Habr and vLLM engine documentation. It is intended for CIOs, CTOs, and business leaders deciding between an external API and deploying a language model on their own servers. This is a practical breakdown: what you will need to build, in what order, and how to verify readiness. We did not test Bitrix24’s infrastructure and are not reproducing its configuration here, because the original article does not include it. AIrassvet is not affiliated with Bitrix24 or the developers of vLLM; the section at the end of the article describes the publisher’s services.
Contents
- In brief: what your own LLM needs besides the model itself
- Why this topic became relevant
- External API or your own model: what changes
- Model card and production: what to check before launch
- Why one shared GPU pool does not work
- Pools by workload profile
- Scheduler, queue, and deferred tasks
- Gateway: permissions, quotas, accounting, and model aliases
- Semantic router and cache router
- Quality Gate: how not to degrade answers through optimization
- What of this does a mid-sized company actually need
- Readiness checklist for a self-hosted LLM
- Frequently asked questions
- Primary sources and verification method
In brief: what your own LLM needs besides the model itself
- Model checks before launch: context at the required concurrency, memory not only for weights but also for KV-cache, API and tool-call compatibility, and licensing.
- Workload-profile pools: fast interactive, long-context, and multi-step agent tasks should be separated, with different limits.
- Scheduler: decides whether to run the request now, put it in the queue, or defer it until capacity is available.
- Gateway: access rights, team and scenario quotas, cost tracking, permanent model aliases.
- Routers: the semantic router chooses the request path, while the cache router selects a replica with a warmed prefix.
- Quality Gate: compare the current configuration and the change against product scenarios before every release.
Why this topic became relevant
On October 12, 2026, Bitrix24 published an article on Habr “Let’s just deploy our own 1T-parameter LLM”: what had to be built around it. The author is Sergey Notevsky, AI Platform Lead at Bitrix24; the material is based on his talk at the Saint HighLoad++ conference in June. According to the author, the company’s AI is used in products across CRM, chats, website generation, and agents: for example, after a call, the model prepares a summary, fills in the CRM record, and evaluates call quality.
The value of this breakdown is that it describes not model selection, but the path “from a product with one endpoint to a platform around models” — something every company faces after the first successful pilot. We covered which models and which hardware to choose for a local deployment separately in the overview “Local LLMs in 2026: models, hardware, and cost”. This article is about what happens after launch.
External API or your own model: what changes
An external API (MaaS, model-as-a-service) means the provider runs the model, and the company pays for tokens. In a self-hosted setup, the model runs on the company’s own infrastructure or in a rented environment, and all operations become its responsibility. According to the author, Bitrix24 started with external APIs and, during the transition, compared options across three axes: data, flexibility, and economics.
| Axis | External API | Your own model |
|---|---|---|
| Data | You need to decide what data can be shared with the provider and where it is processed | Processing happens in your own environment; data requirements also determine acceptable fallback paths |
| Flexibility | The provider defines models, limits, and priorities | Your own selection of models, limits, and priorities tailored to specific use cases |
| Economics | Pay per token; queues, caching, and spikes are hidden behind the API | Capacity is paid for even when idle; queues, caching, and spikes are your responsibility and your cost structure |
We prepared this table based on the breakdown. The author’s key point: at steady load, it makes sense to compare API pricing against infrastructure and team costs, but the move by itself does not mean the token became cheaper. We provided a full total cost of ownership comparison in the article “Cloud or Local LLM: TCO Calculation”.
Model card and production: what to check before launch
It usually starts with a model card on Hugging Face: context, benchmarks, memory requirements. The author shows that every line of the card becomes a separate product question:
| What the card says | What needs to be checked for the product |
|---|---|
| Maximum context length | Does the required context fit at the expected concurrency, and does quality hold up in long scenarios |
| Memory requirements | Are resources sufficient not only for the weights, but also for the KV-cache of simultaneously running requests |
| Tool support | Are the response format, parser, and API contract compatible with your application |
| Benchmark results | Does the model solve your tasks on data similar to production data |
Source of the table — the Bitrix24 article.
KV-cache is the model’s intermediate computations for already processed text, stored in GPU memory while the request is running. The longer the context and the more simultaneous requests there are, the more memory it uses. That is why a model that fits on a GPU by weight may still not handle production traffic.
Three more pre-launch checks the author mentions:
- the inference engine can run the model but not support the required API format or the expected optimization;
- weight format must be matched to the available hardware;
- license: commercial use terms are checked before the product starts depending on the model; the ability to download the weights does not settle that question.
And the main rule for choosing the size is this: if a compact model can handle the job, there is no reason to occupy an expensive compute pool. You need to calculate the cost of the completed task, not a single call: a cheap call helps little if you then have to solve the task again.
Why one shared GPU pool does not work
Imagine a short chat, long document processing, and a multi-step agent task all arriving in the same GPU pool at once. The author explains why they interfere with one another:
- Duration cannot be predicted from the input. The model may answer in a couple of words or go into a long chain of reasoning, and all that time the request occupies memory and a slot among active jobs.
- The concurrency limit is shared. When it is full, the next request waits—even if it is just a short chat reply.
- The number of requests does not equal the amount of work. A single user action may trigger several model calls, and for an agent, the number of steps depends on how the solution unfolds.
Why not just add more replicas? Spare capacity still has to be obtained, the weights loaded, the environment prepared, and the replica verified as ready to accept requests. A spike in interactive traffic can happen faster. That is why capacity reserves need to be in place ahead of time, and they have to be planned with the kind of work that will enter the pool in mind.
Pools by workload profile
Bitrix24 separates load based on what limits execution:
| Pool | What matters | Key characteristic |
|---|---|---|
| Fast interactive | TTFT — time to first token — and response speed | A reserve is needed even if some capacity sits idle |
| Long context | A lot of memory for KV-cache | Limits the allowed concurrency |
| Multi-step tasks | Duration depends on the chain of calls and the amount of generation | The scenario itself sets the latency requirements |
The same model can work in different pools if the scenarios require load isolation: choosing a model and choosing a pool are different decisions.
How many requests can run simultaneously in each pool is tuned based on queue behavior and latency as load increases. The author emphasizes: average GPU utilization does not mean chat is doing fine — you need to look at how long requests wait in the tail of the distribution. And you cannot split pools endlessly either: in one, capacity sits idle; in the next, a queue builds up.
The scheduler, queue, and deferred tasks
The scheduler is the component that receives a request with scenario tags, response-time requirements, and an estimated workload size, then decides whether to run it now, queue it, or defer it. If there is room in the required pool, the task runs; if the limit is busy, it waits; if the result is not needed by the user right now, it is deferred.
Here is how deferred tasks work in the Bitrix24 setup:
- input data and the due date are stored;
- the HTTP connection is not kept open until completion;
- the scheduler returns the task to execution when free capacity appears or the deadline approaches;
- the result is delivered separately.
This creates useful work for capacity between peaks: tasks that can tolerate waiting are handled during less busy hours. But the volume of deferred work is limited with queue length, latency, and each pool’s reserve in mind—otherwise it starts interfering with interactive traffic again. For the business, this is a direct answer to the question in the author’s checklist: "What will you use the capacity for between peaks?" — for example, overnight document processing, call tagging, or reindexing the knowledge base.
Gateway: permissions, quotas, accounting, and model aliases
When several teams and products use a model, every request has an owner. In the Bitrix24 setup, the scheduler is preceded by a gateway with four responsibilities:
- Access: identify the consumer and their permissions.
- Quotas: apply team and scenario limits before sending a request for execution.
- Accounting and tracing: link resource consumption to the request, team, and scenario.
- Aliases: give the product a stable model name behind which the model can be replaced without releasing the product itself.
Accounting is needed not just for reports: it is how teams’ needs are compared, workload profiles are understood, and limits are discussed. A single total expense figure says almost nothing. In smaller projects, ready-made open-source gateways play a similar role—for example, LiteLLM provides a single OpenAI-compatible endpoint, virtual keys with budgets, and expense tracking by team.
Semantic router and cache router
There are two different routers in the architecture, and they should not be confused.
Semantic router chooses the request path before it is queued: a standard answer, a call to a more powerful model, or tool use. The last user message alone is not enough for this: "Yes, let's do it" at the start of a conversation and after a suggestion to perform an action are different requests. The router considers context, intent, whether search or tools are needed, and the expected workload; a follow-up question may mean the previous answer did not fit. We wrote about how model routers work in business scenarios in the article "AI Model Router for Business".
Cache router chooses the replica within the already selected pool. The mechanism is this: if the beginning of a new request matches one that has already been processed, the shared part does not need to be recomputed. According to the vLLM documentation, automatic prefix caching reuses the KV-cache for requests with a shared prefix; this speeds up input processing, but not response generation, and it does not help if the prefixes do not match or the responses are long. Typical high-value scenarios are repeated questions about one long document and long conversations.
A nuance the Bitrix24 author points out: a stable prefix is useless if the next request lands on a different replica that does not have that prefix. That is why the cache router takes into account whether the replica has the needed prefix, how many requests are running on it, and how full its memory is. Sometimes it is better to choose a free replica and recompute the prefix. One more clarification: reused prefix saves compute, but its KV-cache still takes up memory during generation.
Quality Gate: how to avoid hurting response quality with optimization
One way to reduce memory usage is to store the KV-cache at lower precision. The vLLM documentation describes KV-cache quantization to FP8 format: this reduces memory usage, allows more tokens to be held, and supports a longer context. But the calculations change. In the Bitrix24 presentation example, the change stayed within quality requirements on a short context, but not on a long one. The author's conclusion: this kind of decision is made separately for each scenario, and it cannot be judged from latency or memory usage alone.
A Quality Gate is a mandatory comparison of two runs on the same product datasets: the current configuration (baseline) and the proposed change (candidate). According to the author, there are reasons to test even without changing the model:
- updated the engine — the parser, chat template, or dependencies may have changed;
- changed launch parameters, numerical precision, or context handling;
- tweaked the semantic router policy — the request started going down a different branch;
- changed how replies are selected — cache locality and latency changed.
How the testing works:
| Test type | What they test | With what |
|---|---|---|
| Formal | Whether the JSON is valid, whether the values are allowed, whether tool calls are correct | By code |
| Substantive | Match against the reference, alignment with product criteria | By comparison and an LLM judge |
| Operational | Execution latency and cost | By run metrics |
Based on the final diff, they decide whether to roll out the change, test it on limited traffic, or block it. Important detail: product teams add their own scenarios to the dataset, and if the set contains only short queries, you won't learn anything about long context.
What a mid-sized company actually needs from this
The Bitrix24 framework grew out of a product at large scale with many teams. A company launching one or two internal AI features does not need the full stack right away. Below is our editorial staged-growth framework; it is not taken from the Bitrix24 article, but derived from its logic of “add the next block when the previous one is no longer enough.”
| Stage | Indicator | What to add |
|---|---|---|
| 1. One scenario | One function, predictable load | Model validation against the spec and on your own data, a license, basic queue and latency monitoring, a small set of benchmark questions |
| 2. Several scenarios | Chat gets in the way of document processing, complaints about waiting start appearing | Separate pools or limits by load profile, deferred tasks for background work |
| 3. Several teams | It's unclear who is spending how much and why load is increasing | A gateway with permissions, quotas, accounting, and model aliases |
| 4. Agents and different models | Requests need different models and tools | A semantic router, a cache router when there are multiple replicas |
| Always | Any change to the model, engine, or parameters | Quality Gate on product scenarios |
If the load is short and infrequent, the author advises honestly calculating how much time the hardware will sit idle: in that case, an external API or cloud inference for open-source models may be more reasonable. We covered options like this in the article “Open-Source LLM Inference in a Russian Cloud”.
Self-hosted LLM readiness checklist
The first five questions were formulated by the author from Bitrix24, and the rest are an addition from AIrassvet.
- Data. What data requirements limit the choice of processing location? They also determine which fallback paths are acceptable.
- Quality. Can you run a model on the available hardware with the quality your tasks require?
- Utilization. What will you use the hardware for between peaks?
- Team. Who will maintain the models and engines, verify API contracts, tune pools, and investigate regressions?
- Regressions. How will you notice answer quality getting worse after an update?
- License. Does the model license allow your commercial use?
- Metrics. Do you measure time to first token and queue tail wait time, not just average utilization?
- Dataset. Do you have a set of product tasks with ground truth, including long context and tool calls?
- Fallback. What happens to users if a pool or replica goes down? More details in the article on fault tolerance for AI services.
If there is still no answer to questions 2, 4, and 5, your own deployment is premature: start with a pilot on an external API or cloud inference and build a dataset for a future Quality Gate.
Frequently asked questions
Will it be cheaper if we deploy an LLM on our own servers?
Not necessarily. The Bitrix24 author emphasizes that moving on-prem does not make tokens cheaper by itself: the hardware is paid for even when idle, and queues, caching, and handling spikes become the company’s responsibility. You need to compare API pricing with the cost of hardware and the team under your actual workload.
Why not just add more GPUs as the load grows?
A new replica has to be provisioned, loaded with weights, prepared in its environment, and checked for readiness. Interactive load spikes happen faster, so capacity is planned in advance and the load is split across pools.
What is KV-cache in simple terms?
It is the model’s intermediate computation for text it has already read, stored in GPU memory while the request is running. The longer the context and the more simultaneous requests there are, the more memory it uses, which is why it often limits concurrency.
What is prefix cache and when does it help?
It is reuse of computations for a shared request prefix, such as the same instruction or the same document. According to vLLM documentation, it speeds up input processing but not answer generation, and it helps with repeated questions about a long document and in long conversations.
What is a Quality Gate for LLMs?
It is a comparison of the current configuration and the proposed change on the same product datasets, with checks by code, comparison to the reference, and an LLM judge. Based on the results, the change is rolled out, tested on part of the traffic, or blocked.
Where should a company that wants its own LLM start?
With one scenario: test the model on your own data and at real concurrency, read the license, build a set of benchmark tasks, and set up monitoring for queue time and latency. Add pools, a gateway, and routers as the number of scenarios and teams grows.
Primary Sources and Verification Method
We reviewed the primary sources on October 13, 2026. The description of the architecture, design choices, and readiness checklist is based on the Bitrix24 article; the author does not disclose the hardware configuration or metrics, so we do not include them. The information on prefix caching and KV-cache quantization comes from the vLLM documentation. The development-stage table and additional checklist items are AIrassvet’s editorial methodology, not the result of our testing.
- Sergey Notevsky, Bitrix24, Habr, 10/12/2026: “Let’s Just Deploy Our Own 1 Trillion-Parameter LLM”: What Bitrix24 Had to Build Around It — pools, scheduler, gateway, routers, Quality Gate, readiness checklist.
- vLLM: Automatic Prefix Caching — how prefix reuse works and its limitations.
- vLLM: Quantized KV Cache — storing KV-cache in FP8 and its impact on memory.
- LiteLLM: AI Gateway documentation — an example of a ready-made gateway with keys, budgets, and spend tracking.
Limitations: this analysis describes the logical architecture of one large platform without hardware or metric figures; the applicability of each component depends on your workload; vLLM documentation refers to a specific engine and may change between versions.
How AIrassvet Helps You Deploy Your Own LLM as a Service
If you’ve decided to run a language model in your own environment but want to understand in advance which platform components you actually need and how to avoid degrading answer quality while optimizing, it’s worth designing that before you buy the hardware. On the AIrassvet website the following areas are described, all directly relevant to this task:
- Deployment and fine-tuning of language models on company servers, including inference optimization.
- Integrating models into business processes and enterprise AI workspaces.
- Auditing processes and preparing data, including product datasets for quality checks.
- Developing MVPs, testing, launch, team training, and support.
The first step is to choose one use case, describe the current workload and response-time requirements, list data constraints, and define the criterion by which you will consider your model’s answer quality sufficient.
Bottom Line: the Model Is Only the Start of the Platform
Bitrix24’s experience shows that your own LLM becomes a service only when it has separate compute pools around it, a scheduler with deferred tasks, a gateway with quotas and accounting, routers, and a Quality Gate. Every memory or speed optimization can quietly change the answers, so testing on real product scenarios is not optional — it’s a release condition.
Start with one scenario and an honest answer to the checklist: data, quality, load between peaks, team, and how you’ll detect regressions. Add platform components as the previous setup stops being enough, and measure not the token price, but the cost of completing the task.
