For a service used simultaneously by dozens of employees or customers, GPU deployments usually choose vLLM or SGLang: they are designed for parallel requests and splitting a model across multiple GPUs. Ollama is a good fit for a fast start, a prototype, and a small team, while llama.cpp is for running on a CPU, Mac, or underpowered hardware and for quantized models in GGUF format. All four projects are open source and free, and all provide an OpenAI-compatible API, so switching engines usually does not require rewriting the application.
Prepared by AIrassvet October 22, 2026 for CIOs and heads of engineering who are choosing where to deploy an open language model within their own environment. The comparison is based on the projects’ official documentation, README files, and research papers, and the versions and licenses were checked against GitHub on the publication date. We did not run our own load tests for this article: the performance figures belong to the project authors.
Contents
- At a Glance
- What Is and Is Not Included in the Comparison
- What an Inference Server Is and Why You Need One
- Comparison Table
- vLLM: The Production Standard on GPU
- SGLang: Agents, Structured Output, and Shared Prefixes
- llama.cpp: CPU, Mac, and Hybrid Deployment
- Ollama: The Fastest Way to Get Started
- What About TGI and TensorRT-LLM?
- When to Choose What
- How to Test the Choice on Your Own Workload
- Frequently Asked Questions
- Primary Sources and Method
At a Glance
- vLLM — a server for multi-user GPU workloads: PagedAttention, continuous batching, prefix caching, multi-GPU parallelism, and broad hardware support. Apache 2.0 license.
- SGLang — a server in the same class, with an emphasis on agent workflows, structured output, and reuse of shared context (RadixAttention). Apache 2.0 license.
- llama.cpp — a C/C++ engine with no dependencies: CPUs, Apple Silicon, multiple GPU backends, quantization from 1.5 to 8 bits, and hybrid CPU+GPU execution. Built-in server with an OpenAI-compatible API. MIT license.
- Ollama — a convenient wrapper for one-command model deployment on macOS, Windows, Linux, and in Docker. By default, one model handles one request at a time. MIT license.
- Text Generation Inference (TGI) from Hugging Face has been moved into maintenance mode, and its GitHub repository is archived; for new projects, the authors themselves recommend vLLM, SGLang, and llama.cpp.
What Is and Is Not Included in the Comparison
Included in the comparison: the purpose of the four open inference servers, supported hardware, handling parallel requests, API support, quantization, multi-GPU distribution, monitoring, licenses, and selection criteria.
Not included in the comparison: choosing the model itself and sizing the hardware (see the overview of local LLMs, hardware, and costs), comparing cloud and on-prem servers by total cost, and paid enterprise platforms. We do not publish our own speed numbers: they depend heavily on the model, GPU, prompt length, and settings.
What an Inference Server Is and Why You Need One
An inference server is a program that loads a language model into memory and answers application requests via API. The model is the weights, and the server determines how fast those weights will run and for how many users at the same time.
The main technical difference between servers is how they manage memory for the context cache (KV cache) and batch incoming requests. The cache stores intermediate results for each active conversation and grows with text length. A server that efficiently lays out this cache and combines requests into batches on the fly can serve more users on the same GPU. The authors of vLLM report in the PagedAttention paper a 2–4x increase in throughput at the same latency compared with FasterTransformer and Orca.
For business, the choice of server determines three things: how many GPUs you need for the workload, how easy the service is to operate, and what hardware you can run on at all. We covered what this looks like at the platform level in the article “Your Own LLM in the Company: Queues, GPU, and Quality Control”.
Comparison Table
| Criterion | vLLM | SGLang | llama.cpp | Ollama |
|---|---|---|---|---|
| Primary use case | Multi-user service on GPU | Multi-user service, agents, structured output | Local and embeddable deployment on any hardware | Fast model deployment on a workstation or server |
| License | Apache 2.0 | Apache 2.0 | MIT | MIT |
| Latest release as of 10/22/2026 | v0.32.0, October 18, 2026 | v0.5.22, October 19, 2026 | v0.7.0, October 15, 2026 | v0.40.8, October 21, 2026 |
| Hardware per documentation | NVIDIA, AMD, and Intel GPUs; x86, ARM, and PowerPC CPUs; plugins for TPU, Gaudi, Ascend, Apple Silicon, and others | NVIDIA and AMD GPUs; TPU; Intel GPU and Xeon; Apple Silicon; Huawei Ascend; Moore Threads | x86, ARM, and RISC-V CPUs; Apple Silicon; CUDA, HIP, MUSA, Vulkan, SYCL, CANN (Ascend), and other backends | macOS, Windows, Linux, Docker |
| Parallel requests | Continuous batching, PagedAttention | Continuous batching, RadixAttention | Server slots (--parallel), continuous batching enabled by default |
OLLAMA_NUM_PARALLEL, default is 1 request per model |
| Multiple GPUs | Tensor, pipeline, data, expert, and context parallelism | Tensor, pipeline, expert, and context parallelism (launch parameters) | Default layer-wise splitting, row-wise splitting, tensor mode — experimental | Not described in the verified documentation |
| API | Compatible with OpenAI, Anthropic Messages API, gRPC | Compatible with OpenAI | Compatible with OpenAI (llama serve) |
Own REST API and OpenAI-compatible /v1 |
| Quantization | FP8, INT8, INT4, GPTQ, AWQ, GGUF and others | Pre-quantized weights (GPTQ, AWQ, etc.) and online quantization, including FP8 | Integer from 1.5 to 8 bits, GGUF format | Ready-made models from the ollama.com/library catalog |
| Monitoring | Endpoint /metrics for Prometheus |
Prometheus metrics via the --enable-metrics |
--metrics, compatible with Prometheus |
Built-in metrics are not described in the verified documentation |
| Entry barrier | Medium: Python or Docker, GPU-specific setup | Medium: Docker or Python | Low for a single user, higher for fine-tuning | Minimal: installation and one command |
Versions and licenses were obtained via the GitHub API on October 22, 2026; capabilities are from the projects' README files and documentation as of the same date (vLLM metrics, SGLang metrics, SGLang quantization, SGLang launch parameters). The feature set changes from version to version, so before choosing, check the version you plan to install.
vLLM: the production standard for GPU inference
vLLM is an open-source server for multi-user workloads that started with a research paper on PagedAttention. The README lists its core mechanisms: PagedAttention for cache memory management, continuous batching of incoming requests, chunked prefill and prefix caching, as well as tensor, pipeline, and other forms of parallelism for distributing the model across multiple GPUs.
What matters for business:
- Compatibility. The server exposes an OpenAI-compatible API, as well as Anthropic Messages API and gRPC. An app built for a cloud API can be switched over by changing the endpoint.
- Hardware. In addition to NVIDIA GPUs, AMD and Intel GPUs, x86, ARM, and PowerPC CPUs, and plugins for Huawei Ascend, Google TPU, Intel Gaudi, and Apple Silicon are also supported. This matters if GPU procurement is limited.
- Quantization. FP8, INT8, INT4, GPTQ, AWQ, GGUF, and other formats are supported; we wrote about the trade-offs between memory and quality in the article “LLM Quantization: What It Is”.
Limitation: vLLM is designed for server-class GPUs and requires tuning for the model and workload. For a developer laptop or a single user, it is overkill.
SGLang: agents, structured output, and shared prefixes
SGLang is a server in the same class as vLLM, described in the README as optimized for agent workloads, large-scale serving, and RL training. In the authors' paper two key mechanisms are described: RadixAttention, which reuses the cache for the shared beginning of requests, and compressed finite-state automata for faster generation of structured output, such as JSON. The authors report throughput up to 6.4x higher than other systems on agent orchestration, JSON generation, RAG, and multi-step dialogue tasks.
This shows up in practice when AI agents and support chatbots repeat a long shared system prompt and tool instructions in every request. A server that does not recompute that shared prefix from scratch saves compute.
On hardware, SGLang lists NVIDIA GPUs (from the RTX 30 series to B300), AMD Instinct, Google TPU, Intel, Apple Silicon, as well as Huawei Ascend and Moore Threads. The package also includes SGLang Diffusion, an image and video generation engine.
Limitation: the project is moving very fast, and the feature set depends on the platform. Pin the version and run regression tests before upgrading.
llama.cpp: CPU, Mac, and hybrid execution
llama.cpp is a pure C/C++ engine with no dependencies, designed to run models with minimal setup across a wide range of hardware. The README claims optimization for Apple Silicon, AVX, AVX2, AVX512, and AMX instructions for x86, integer quantization from 1.5 to 8 bits, custom CUDA kernels, AMD support via HIP, Moore Threads via MUSA, Vulkan, SYCL, and Huawei Ascend via CANN.
The most business-relevant feature is hybrid CPU+GPU execution: part of the model runs on the GPU, and the rest runs on the CPU. This makes it possible to run a model that does not fit entirely in video memory, at the cost of speed.
The built-in server (llama serve) exposes an OpenAI-compatible API. Its documentation includes settings for multiple concurrent slots, continuous batching (enabled by default), prompt caching, Prometheus-format metrics, and model-splitting modes across multiple GPUs, with tensor mode currently marked as experimental.
Limitation: llama.cpp mainly works with its own GGUF weight format, and for high multi-user load on server GPUs, vLLM or SGLang are usually chosen. A good niche is offline environments, workstations, Macs, and servers without GPUs.
Ollama: the fastest way to get started
Ollama turns model deployment into a one-command setup: installers for macOS, Windows, and Linux, an official Docker image, a library of ready-made models, a custom REST API on port 11434, and OpenAI-compatible endpoints /v1.
Parallel execution is configured through environment variables described in the Ollama FAQ:
OLLAMA_NUM_PARALLEL— how many requests one model handles at the same time; the default is 1; memory usage grows in proportion to this number multiplied by the context length;OLLAMA_MAX_LOADED_MODELS— how many models can be loaded at once; the default is 3 per GPU or 3 when running on CPU;OLLAMA_MAX_QUEUE— how many requests will be queued; if the queue overflows, the server returns a 503 error.
That is where Ollama fits: prototypes, internal tools for a small team, and developer and analyst workstations. If the service needs to handle dozens of simultaneous users, after the pilot it is usually moved to vLLM or SGLang — thanks to the compatible API, that means changing the endpoint and parameters, not rewriting the application.
One separate security note: Ollama can also work with cloud models after signing in to an account. If data must not leave the perimeter, use only local models and block the server’s internet access.
What about TGI and TensorRT-LLM?
Text Generation Inference from Hugging Face was long one of the main options. Its README now says the project has been moved to maintenance mode and only accepts minor fixes, and the authors themselves recommend vLLM, SGLang, and local engines like llama.cpp and MLX. According to the GitHub API as of October 22, 2026, the repository is archived. TGI is not a good choice for a new project, and existing deployments should be planned for migration; more on that process in the article about migration when models are decommissioned.
TensorRT-LLM — NVIDIA’s library for optimized inference on NVIDIA GPUs. It makes sense when the infrastructure is built entirely on NVIDIA hardware and maximum performance is needed at the cost of a more complex build. We do not cover it in detail in this comparison.
When to choose what
| Situation | Recommended choice | Why |
|---|---|---|
| Pilot or prototype, 1–5 users | Ollama | Lowest barrier to entry, compatible API |
| Server without a GPU, offline environment, Mac | llama.cpp | Works on CPU and Apple Silicon, GGUF quantization, hybrid mode |
| The model does not fit in VRAM, and there is only one GPU | llama.cpp | Some layers run on the CPU |
| Enterprise assistant or API for dozens or hundreds of users on GPU | vLLM | Continuous batching, PagedAttention, multi-GPU parallelism |
| AI agents, long shared prompts, JSON generation | SGLang or vLLM | RadixAttention and structured output acceleration in SGLang, prefix cache in vLLM; choose by testing |
| GPUs that are not NVIDIA (AMD, Ascend, Moore Threads) | Test vLLM, SGLang, and llama.cpp on the specific model | All three claim support, but the feature set depends on the platform |
| TGI is already in use | Plan a migration to vLLM or SGLang | TGI is in maintenance mode, repository archived |
A typical company path is to start with Ollama or llama.cpp during the pilot stage, then move to vLLM or SGLang when going into production. If you are comparing not the engines but the hosting option itself, see the breakdown of cloud vs. local LLM total cost.
How to validate the choice on your own workload
Figures from articles and third-party benchmarks were obtained on different models, GPUs, and requests. It is better to confirm the decision with a short test:
- Fix the model and weight format. The same model and quantization variant for all candidates, otherwise you are comparing different things.
- Build a workload profile. Typical prompt and response length, number of concurrent users at peak, share of requests with a long shared prompt.
- Run 2 candidates on one server. For example, Ollama and vLLM, or vLLM and SGLang.
- Measure three metrics. Time to first token, generation speed per user, and total throughput at peak concurrency.
- Check operations. Metrics, logs, restart, version upgrade, queue handling under overload.
- Check answer quality. Using your own set of requests, make sure the responses from the two servers are equal in quality: different settings and quantization can affect the result.
Frequently asked questions
How is vLLM different from Ollama?
vLLM is a server for multi-user workloads on GPUs: continuous batching, PagedAttention, and multi-GPU parallelism. Ollama is a tool for quickly launching models with a single command, where by default one model handles one request at a time. Ollama is more convenient for a pilot, while vLLM is better for a production service.
Can you run an LLM without a GPU?
Yes, llama.cpp is the best fit for that: it is optimized for x86 and ARM CPUs and Apple Silicon and supports quantization from 1.5 to 8 bits. Performance on CPU is noticeably lower, so this option is suitable for small models, a small number of users, or offline environments where isolation matters more than speed.
Will the application need to be rewritten when changing servers?
Usually not. vLLM, SGLang, llama.cpp, and Ollama provide an OpenAI-compatible API, so the application switches by changing the server address and model name. You should check for differences in support for individual parameters, tool calling, and structured output, which vary between servers and versions.
Which server should you choose for AI agents?
For agents, the important things are reusing a long shared prompt and quickly generating JSON for tool calls. These scenarios are directly optimized by SGLang (RadixAttention and structured output acceleration), and vLLM supports prefix caching. The best choice between them should be made with a load test on your own agent workflows.
Primary sources and methodology
- Project README files and documentation: vLLM, SGLang, llama.cpp and llama-server documentation, Ollama, Ollama FAQ and Ollama compatibility with OpenAI, Text Generation Inference.
- Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention, arXiv:2309.06180.
- Zheng et al. SGLang: Efficient Execution of Structured Language Model Programs, arXiv:2312.07104.
- Versions, release dates, licenses, and archive status were obtained through the GitHub API on October 22, 2026.
Method: we compared the projects using the same set of criteria based only on primary sources and did not run our own load tests. The performance figures are the authors' and were obtained under their own conditions.
How AIrassvet helps deploy an LLM on a company server
Choosing an inference server is one of the decisions when launching your own model, alongside choosing the model, hardware, and quality control. On the AIrassvet website the following areas are listed as addressing this task:
- LLM implementation and deployment, including on the customer’s servers: selecting the model and inference server based on the workload and available hardware;
- integrating the model with enterprise systems and AI agents through a compatible API;
- testing and launch: load testing candidates, checking answer quality, and post-launch support.
The first step is to choose one process that needs its own model and document the current result, data sources, constraints, and acceptance criteria.
Bottom line: the choice of server determines the number of users and the hardware
In short: Ollama is for getting started and small teams, llama.cpp is for CPUs, Mac, and offline setups, vLLM is for multi-user GPU-backed services, and SGLang is for agent workflows and structured output with the same server load. TGI is no longer suitable for new projects.
All four projects are free and compatible with the OpenAI API, so you can start with the simple option and move to a server-based setup as the workload grows. Make the final decision based on a short test with your model, hardware, and request profile.
