How to choose an AI development company: 12 questions to ask before you sign

A demo proves that a model can produce a good answer once. A production AI system has to produce good answers on inputs nobody planned for, at a cost you can sustain, without leaking data or taking actions it should not. The difference is almost entirely in the engineering around the model, and that is what these questions probe.

About your data

1. What data will the system need, and how will it get it? A good partner asks where your information lives before talking about models. If the answer is 'in people's inboxes', the first project is data access, not AI.
2. How will you keep it current? Retrieval systems go stale. Ask how new and changed documents reach the index, and how deleted ones leave it.
3. How will you enforce who can see what? If an employee cannot open a document today, the AI must not quote it to them tomorrow. Permission-aware retrieval should be in the design, not a later phase.

About accuracy

1. How will we measure whether it works? The right answer involves an evaluation set built from your real cases, with expected outputs, run on every change. 'We will test it' is not an answer.
2. What happens when the answer is not in the data? You want a system that says it does not know. Ask to see how that behaviour is tested.
3. How do we catch regressions when the model or prompt changes? Vendors update models frequently. Without automated evaluation, quality can drop silently.

About cost

1. What will it cost to run per month at our volume? Expect an estimate based on token counts, caching and model choice, not just the build fee. Our AI cost calculator shows the variables.
2. Which model, and why? A credible partner compares at least two models on your evaluation set and can explain the trade-off between quality, speed and price.
3. Can we switch models later? Lock-in to one provider is a business risk. Ask whether the design isolates the model behind an interface.

About safety and ownership

1. Will our data be used to train anyone's model? The answer should name the API tier or deployment that guarantees it will not.
2. If the system takes actions, what stops a bad one? For agents: least-privilege tools, approval steps for irreversible actions, spending limits and a complete action log.
3. Who owns the code, prompts and evaluation set? You should, from day one, in your own repository and cloud accounts.

Red flags

• A fixed price for a full AI build before anyone has looked at your data.
• Accuracy promises such as '99%' with no evaluation method behind them.
• Case studies with impressive numbers but no way to verify any of them.
• No mention of running costs, monitoring or what happens after launch.

Comparing specialists? Browse AI and machine learning companies in the CrpyByte directory, or ask us the same 12 questions.