On-prem LLM costs: when self-hosting beats paying per token

The honest answer to "is on-prem cheaper than the API" is that it depends on one number, and most people have not measured it: how many tokens per month you actually put through the model in production.

Below that number, self-hosting is more expensive and more work. Above it, the gap widens every month. This is how to find where your workload sits before committing to hardware.

The shape of the two cost curves

Commercial APIs are pure variable cost. Zero volume costs zero, and every additional token costs the same as the last one. The line goes through the origin and never bends.

Self-hosting is mostly fixed cost. The GPU, the power, the rack space and the engineer who keeps it running cost the same whether you serve a thousand requests a day or a million. The line starts high and stays nearly flat.

Two lines like that cross exactly once. Everything else in this article is about locating the crossing point and being honest about what sits on each side of it.

What the API side actually costs

Take the last three months of billing, not a projection. Then add the parts that are easy to leave out:

• •Input and output tokens priced separately, with output typically several times more expensive — a summarisation workload and a generation workload with the same request count have very different bills.
• •Retries. Failed and re-run requests are billed.
• •System prompts and few-shot examples, resent with every single call. On a long system prompt this is frequently the largest line item and the one nobody counts.
• •RAG context. Retrieved chunks are input tokens, and a generous top-k multiplies them.
• •Growth. Compare against where volume will be in twelve months, not where it is now.

The system prompt is the most commonly missed cost. A 2,000-token system prompt on 100,000 calls a month is 200 million input tokens before the user has typed anything.

What the on-prem side actually costs

The GPU price is the number people quote and the smallest part of the real figure.

Engineering time is where most self-hosting business cases quietly fail. A GPU serving production traffic is infrastructure: it needs monitoring, an on-call path, patching, and someone who understands why throughput dropped after a driver update. Costing that at zero because "we already have a DevOps person" is the same error as costing a database at zero for the same reason.

Working out the crossing point

The arithmetic is simple once the inputs are honest:

monthly API cost   = (input tokens x input price) + (output tokens x output price)
monthly on-prem    = hardware/36 + power + hosting + engineering time

crossover reached when: monthly API cost > monthly on-prem

Amortise hardware over 36 months, not 12. GPUs do not stop working after a year,
and a 12-month amortisation makes on-prem look artificially bad.

Then apply a utilisation check, which is the step that gets skipped. A GPU costs the same whether it is at 5% or 85%. If traffic is a spike for two hours a day and idle the rest, the effective cost per token is far worse than the headline calculation suggests. Steady, predictable load is what makes self-hosting win; bursty load is what the API is genuinely good at.

Where each side wins

A worked example from our own client

Nocodo LTD came to us with an OpenAI API bill that had reached $10,000 a month and was growing with usage. We deployed an open-source model on infrastructure they control. Their spend went to $2,000 a month — an 80% reduction, roughly $96,000 a year — with API responses running about three times faster, and the whole thing took four weeks from start to a working endpoint.

Why it worked there, stated plainly so you can check whether it applies to you: their volume was high, it was steady rather than spiky, the workload did not need frontier reasoning, and they had a compliance reason to want the data on their own infrastructure anyway. Change any one of those and the arithmetic changes with it.

An 80% reduction is not a rate card. It is what one workload with a particular shape produced. A company spending $800 a month on bursty, low-volume calls would spend more after migrating, not less, and we would tell them so.

The technical choices that move the number

Model size

The instinct is to pick the largest model that fits. The better question is the smallest model that passes your evaluation set. Model size drives VRAM, VRAM drives GPU class, and GPU class drives the entire fixed cost. Dropping from a 70B to a well-chosen smaller model, or to a quantised version of the same one, frequently changes which hardware tier you need at all.

Quantisation

Running at reduced precision cuts VRAM substantially with quality loss that is often immaterial for a specific task, though never zero. The only way to know whether it is acceptable is to measure on your own workload rather than to read a benchmark table produced on someone else's.

The inference server

This is not a detail. Ollama is excellent for getting something running and for development. For production concurrency, vLLM's continuous batching serves far more simultaneous requests from the same card, which changes the cost per token directly. Putting LiteLLM in front gives an OpenAI-compatible endpoint, so application code usually needs no changes beyond a base URL — and it keeps the option of falling back to a commercial API for the requests that genuinely need one.

# what to measure before buying anything, on a rented GPU by the hour
# 1. does the model pass your eval set at the quantisation you plan to use?
# 2. tokens/sec at your real concurrency, not at concurrency 1
# 3. VRAM headroom at peak context length

vllm serve <model> \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 64        # concurrency is the number that decides your hardware

Rent before you buy

Nothing above requires owning a GPU to test. Rent one by the hour, run your actual workload against it for a week, and measure throughput at real concurrency. That week costs very little and answers the two questions that matter: whether the open model is good enough for your task, and how many tokens per second one card genuinely delivers under your load.

Only then does the hardware conversation make sense, because only then do you know how many cards the number implies.

Reasons to move that are not cost

Cost is the argument that gets budget approved, but it is often not the real driver:

• •Data residency and compliance — prompts and completions never leaving your boundary is a categorical difference, not a cheaper one.
• •Predictability — a fixed monthly figure instead of a bill that scales with a product launch.
• •No vendor deprecation — the model you validated stays exactly as it was, instead of shifting under you.
• •Latency — no round trip, and no shared-tenancy queue at peak times.

Any of these can justify self-hosting at volumes where the pure cost case does not. They should be argued on their own terms rather than dressed up as savings.

How we scope this

Our assessment is deliberately boring: measure current token volume and its shape, run your workload against candidate models on rented hardware, size for concurrency rather than for the model, and produce the crossover figure with the assumptions written down so you can challenge them. If the arithmetic says stay on the API, that is the deliverable, and it is a cheaper answer to receive than to discover after buying a GPU.