12 Questions Procurement Should Ask Every AI Vendor

Software procurement teams know their craft, but AI purchases hide risk in new places. These twelve questions surface it early. The wrong answers are listed too, because vendors have learned to sound compliant.

Capability and evidence

1. "Show accuracy on our data, not your benchmark." Any serious vendor supports a structured pilot on your labelled examples. Refusal, or benchmark-only answers, means the demo is the product.

2. "What is the failure mode, and how do we see it?" You are buying the error handling. Ask for the dashboard where wrong answers surface and the process that fixes them.

3. "Which model providers underlie this, and what happens when they change?" Wrappers are fine — undisclosed wrappers are not. Model deprecations and behaviour shifts should be their managed problem, contractually, not your surprise.

4. "Can we see the evaluation suite?" Vendors who test seriously are proud of it. Vendors who "iterate based on feedback" are testing on your customers.

Data and law

5. "Is our data used for training, and where is that written?" The answer must be in the contract, not the sales thread. EU customers: ask where inference runs and where logs live.

6. "How do you handle deletion requests that touch model behaviour?" Tests whether they understand their own privacy obligations. Blank looks here predict blank looks at audit time.

7. "What are your EU AI Act obligations for this product, and which are ours?" In 2026 a vendor without a crisp provider-vs-deployer answer is transferring their homework to you, with penalties attached.

Money and exit

8. "Price this at 10x our pilot volume." Per-seat AI pricing often hides per-token reality. Get the curve in writing.

9. "What do we keep if we leave?" Prompts, configurations, fine-tuned weights, labelled data, logs. "Nothing" is an answer; it should be priced accordingly.

10. "What breaks in month one after cancellation?" Maps the real dependency before it maps you.

Delivery honesty

11. "Who exactly does the implementation?" The pitch team and the delivery team are different companies wearing one logo more often than anyone admits.

12. "Tell us about a deployment that failed." Everyone has one. A vendor who cannot name one is either new or lying, and both are information.

A vendor who answers all twelve cleanly is rare and worth shortlisting. We sit on the buyer's side of this table for clients as a fixed-scope advisory engagement — including running the bake-off pilot so the decision is made on your data, not their slides.