Buying GuidesGuide
Self-hosted LLM or a model API? How to decide on cost, privacy and effort
Running your own model sounds cheaper and more private. Sometimes it is. Often the API wins once you count the GPU hours you pay for while nobody is using them, and the engineer who keeps it running.
Author
CrpyByte Engineering
Published
Reading time
3 min read
There are two ways to put a language model into your product. You can call a hosted API and pay per token, or you can run an open-weight model yourself on a GPU server you rent or own. Both are reasonable. The right answer depends on four things: volume, data rules, the quality you need, and who will maintain it.
What you actually pay for in each option
| Model API | Self-hosted model | |
|---|---|---|
| Main cost | Tokens in and out, per request | GPU server time, whether it is busy or idle |
| Scales with | Usage | Capacity you reserve |
| Hidden cost | Long prompts and large retrieved context | Engineering time: updates, monitoring, outages |
| Quiet month | Low bill | Same bill |
| Busy month | Higher bill, no capacity work | Same bill until you need more GPUs |
The key difference is shape. An API bill follows your usage. A self-hosted bill is mostly fixed. Self-hosting only becomes cheaper when the server is busy most of the time. A GPU server that answers a few hundred questions a day spends most of its life idle, and you pay for every idle hour.
When self-hosting is the right call
- Data must not leave your infrastructure. Contracts, regulation or client promises rule out sending the text to a third party, even one with strong data terms.
- High, steady volume. Large batch jobs such as classifying, extracting or transcribing thousands of documents a day can keep a GPU busy enough to beat per-token pricing.
- The task is narrow. Speech-to-text, embeddings, tagging and extraction run well on smaller open models. You do not need the largest model for them.
- You need it offline or at a fixed location, for example on a factory network or in a building with no reliable internet.
- You already have the people. Someone on the team can patch drivers, watch memory, and restart services at 2 AM.
When an API is the better buy
- Usage is low or uneven. A support chatbot, an internal assistant or a pilot rarely keeps a GPU busy.
- You need the strongest reasoning. The best general-purpose models are usually available only through APIs.
- Speed of delivery matters. An API integration ships in days. A production GPU deployment takes longer and needs its own monitoring.
- Nobody owns the infrastructure. Without a person responsible for it, a self-hosted model becomes the system that fails when that person is on holiday.
The privacy question, answered precisely
"Private" is not one setting. Ask which data is not allowed to leave, then pick the option that keeps exactly that data in. Many teams end up with a mix: sensitive steps such as transcription or document search run on their own server, and general reasoning on anonymised or non-sensitive text goes to an API. Read the provider's data terms for retention, training use and region before deciding an API is ruled out.
A quick decision table
| Your situation | Usually best |
|---|---|
| Pilot or proof of concept | API |
| Customer chatbot, moderate traffic | API with a spending cap |
| Sensitive documents that must stay in-house | Self-hosted for those steps |
| Thousands of documents a day, narrow task | Self-hosted smaller model |
| Mixed sensitive and general work | Hybrid: self-host the sensitive part |
Before you buy GPU hardware
- Estimate monthly tokens for your real use case and price it on an API first. The free AI cost calculator and token counter help.
- Compare with the full self-hosted cost: the server, storage, backups, and a realistic share of an engineer's time.
- Test quality on your own data with both a hosted model and the open model you would run. A cheaper model that fails your test set is not cheaper.
- Plan for change. Models are replaced every few months. With an API, you change one setting; self-hosted, you download, test and redeploy.