AI & AgentsGuide
What an LLM application really costs: build, run and maintain
Most AI budgets focus on the build and underestimate everything after launch. Here is how the cost of a production LLM application actually breaks down, and the levers that reduce each part.
Author
CrpyByte Engineering
Published
Reading time
2 min read
An LLM application has three cost layers: the one-off build, the monthly running cost, and the ongoing work to keep it accurate. Buyers usually ask about the first and discover the other two later. Planning all three up front is what keeps an AI project profitable.
1. The build
The model call is a few lines of code. The build effort goes into everything around it:
- Data ingestion: connectors to your documents or systems, cleaning, chunking and keeping it in sync.
- Retrieval: embeddings, a vector or hybrid search index, and permission filtering.
- Application: the interface or integration your users actually touch, plus authentication and logging.
- Evaluation: a labelled test set from your real cases and the harness that runs it on every change.
Evaluation is the line item most often cut, and the one whose absence causes most production failures. Budget for it explicitly.
2. Running costs
Monthly cost is driven by volume multiplied by tokens per request multiplied by price per token, plus infrastructure.
| Driver | What raises it | How to lower it |
|---|---|---|
| Input tokens | Long prompts, large retrieved context, full chat history | Retrieve fewer, better chunks; summarise history; prompt caching |
| Output tokens | Verbose answers | Ask for structured, concise output; set max tokens |
| Model tier | Using the largest model for every request | Route simple requests to a smaller model |
| Retrieval infrastructure | Large indexes, managed vector DB tiers | Right-size; Postgres + pgvector is enough for many cases |
| Batch vs real-time | Running offline jobs through real-time APIs | Use batch APIs for non-urgent work, usually at a discount |
You can model these numbers for your own volume with the free AI cost calculator and token counter.
3. Keeping it accurate
Models change, your documents change, and users ask questions nobody anticipated. Plan for a small, steady maintenance effort:
- Reviewing logged failures and adding them to the evaluation set.
- Re-running evaluation when the provider updates or retires a model.
- Updating prompts, retrieval settings and connectors as your data evolves.
- Monitoring cost per request so a change does not silently double the bill.
A sensible order of spending
- A short feasibility sprint on real data, to learn whether accuracy is achievable and what running cost to expect.
- A focused first release for one use case, with evaluation and monitoring included.
- Expansion to further use cases once the first one has proven its value in production.