Ollama vs vLLM in production: the decision, not the benchmark

The two get compared as if you would choose between them for the same job. You mostly would not. Ollama exists to make running a model on one machine trivial; vLLM exists to serve a model to many callers at once, fast. Almost every "which is faster" argument is really two people describing different workloads and not saying so.

Here is the decision as it actually presents itself when you are standing up a self-hosted endpoint.

What each one is built for

Ollama packages a llama.cpp-derived engine behind a friendly CLI and API, pulls quantised models by name, and runs on a laptop or a modest GPU with almost no setup. It will happily handle a handful of concurrent requests, and recent versions parallelise them, but that is not what it is optimised for — it is optimised for getting a model answering on your own hardware in about five minutes.

vLLM is an inference server. Its whole design — PagedAttention to manage the KV cache efficiently, and continuous batching to keep the GPU busy across many in-flight requests — exists to push the highest possible throughput through a GPU under concurrency. It is GPU-first, needs more setup, and rewards you when the endpoint is genuinely busy.

The number that decides it is concurrency

Everything follows from how many requests hit the endpoint at the same time.

At low concurrency the two feel similar and Ollama is less work. As concurrent requests climb, vLLM pulls away sharply, because continuous batching interleaves many requests through the model instead of serving them one after another, and PagedAttention stops the KV cache from fragmenting memory the way naive serving does. That is the entire reason vLLM exists, and the entire reason a busy endpoint on Ollama feels like it hits a wall.

Where each is the right answer

Reach for Ollama when the workload is a handful of users or an internal tool, when you are prototyping and want to swap models by name, when you need a CPU fallback, or when "an engineer can stand it up this afternoon" matters more than squeezing the GPU. It is a genuinely good default for everything that is not a busy production endpoint.

Reach for vLLM when the endpoint serves real concurrent traffic, when you have latency or throughput targets to hold, and when the GPU is expensive enough that its utilisation is a line item you care about. The setup cost is real; you take it on because the alternative is buying more GPU than you need to compensate for serving inefficiency.

The part the comparison usually skips

Both expose an OpenAI-compatible API now, so the application code rarely cares which one is behind the endpoint — you can prototype on Ollama and move to vLLM for production without rewriting the client. That makes the choice less permanent than it looks, and argues for not agonising over it early.

What neither tool gives you is the operational layer around it: monitoring, an on-call path, VRAM sizing for your actual model and context length, and a plan for when a driver update drops throughput. Choosing the serving engine is the easy decision. Sizing the hardware honestly is the one that decides the bill — which is a separate question we worked through in when self-hosting an LLM beats paying per token.

How this played out for a client

For Nocodo LTD the workload was steady, concurrent production traffic that did not need frontier reasoning — the shape where a self-hosted, batched serving engine earns its keep. Moving off a per-token API cut their spend by around 80% with faster responses. The point is not that self-hosting always wins; it is that the workload's shape, concurrency included, is what tells you which tool — and whether to self-host at all — is right.