Data Science Vendor Evaluation Checklist: Score Providers on Delivery Risk
What matters most about scoring data science vendors
- Score evidence, not confidence. A 3 or 4 on any criterion should require something you can check, such as a named person, a test plan or a contract clause, so a polished pitch with no specifics lands at 1.
- Seven weighted criteria cover data access, delivery, test method, ownership, cost, references and risk. In the default weighting, cost carries 10 of 100 points and delivery and testing carry 40 between them.
- Fix the weights before you open any proposal, then re-run the total with alternative weights. If the ranking flips, your team disagrees about priorities, and that is worth settling before you sign.
- Four pass/fail gates come before scoring: data-handling terms, ownership position, a named delivery lead and a stated success test.
A data science vendor evaluation checklist is only useful if two people reading the same proposals would reach similar scores. Most comparisons fail on that test. One reviewer is impressed by a demo, another by the lowest price, and the final choice rests on whoever argues best. This article gives you a form to avoid that: gates, a 0–4 evidence scale, default weights you can adjust, a worked example with two hypothetical vendors, and a printable checklist at the end. It does not name or rank providers; if you need a shortlist first, start with our guide to the best data science providers. This checklist is built for the proposals already on your desk.
Who should use this data science vendor scorecard (and when it fits poorly)
The scorecard suits a procurement manager, or a project owner working with procurement, who has two or more written proposals for a defined piece of data science work: a forecasting model, a churn or scoring model, an analytics build, or a proof of concept. It works best when the proposals describe a project with a start, an end and some deliverables you can inspect.
It fits less well in two situations. An open-ended retainer has no acceptance test to score, so you would be judging a team rather than a project, and the test-method criterion below would need rewriting around team performance. A single-contractor engagement shrinks several criteria, such as team depth and governance, to near-trivial answers. If you are still working out what kind of partner you need, read our guide on how to choose a data science consulting partner first, then come back to scoring when the proposals arrive.
Start with four pass/fail gates
Gates keep a weighted total from hiding a deal-breaker. A vendor can score well on seven criteria and still be unusable if it will not accept basic data-handling terms. Check these before any scoring begins:
- Data-handling terms. The vendor will accept written terms on where your data is processed, who can access it, and how it is deleted afterwards. If personal data is involved, data protection law may require specific written terms with a processor, so involve your legal or data protection lead early.
- Ownership position. The proposal says in writing who will own the deliverables (code, trained models, documentation) and which parts of the vendor’s own tooling you are only licensed to use.
- Named delivery lead. A specific person is accountable for delivery, and the proposal says who will do the work.
- A stated success test. The proposal says how anyone will judge that the work is good enough, even if the details are still to be agreed.
A vendor that fails a gate is not necessarily out. Send a clarification question and see whether the answer arrives in writing. If a vendor will not put a gate item in writing before it has your business, assume it will not do so afterwards either.
The seven criteria and the evidence that earns a score
Each criterion is scored from 0 to 4. The scale is the same for all seven, so evaluators do not have to learn seven different grading systems:
| Score | Meaning |
|---|---|
| 0 | Missing, or contradicted elsewhere in the proposal. |
| 1 | Claimed but without detail or evidence, such as “we follow best practice”. |
| 2 | Described specifically, but not yet checkable or only partly complete. |
| 3 | Specific and checkable through a document, person, demo or clause. |
| 4 | Specific, verified and committed in the draft contract or statement of work (SOW). |
| Criterion | What you are scoring | Evidence that earns a 3 or 4 |
|---|---|---|
| 1. Data access and security handling | How the vendor will reach your data: where the work happens, who gets access, what is copied, how long it is kept and who else touches it. | A written data-handling plan, a named working environment, access limited to named roles, deletion on completion written into the SOW, and a list of subcontractors that will see the data. |
| 2. Delivery plan and team | Whether the plan can actually be run: milestones, what the vendor needs from you, and who does the work. | Named lead and roles with time allocations, milestone dates with acceptance criteria, and a dated list of inputs you must supply. |
| 3. Test method and acceptance criteria | How anyone will know the work is good enough. | A baseline to beat (even a simple one, like last period’s value), a validation approach using data the model has not seen, a metric tied to the business decision, an agreed pass mark and a stated consequence if it is missed. |
| 4. IP, ownership and exit | Who owns code, models, features and documentation, what you are only licensed to use, and whether you could run the result without the vendor. | An ownership and licence clause, a list of vendor-owned and third-party components, and a handover package covering code, documentation and a reproducible environment. |
| 5. Cost and commercial terms | Total cost under stated assumptions rather than the headline figure. | An itemised estimate, written assumptions, change control, payments linked to milestones, and a clear list of exclusions such as cloud usage, licences and support. |
| 6. References and evidence | Proof of comparable work. | A reference you can actually contact, work of similar scale and type, and a demo run on a sample of your data rather than a prepared dataset. |
| 7. Risk and governance | Third-party components, model risk, incident handling and monitoring after handover. | Disclosure of any external models or APIs used, a breach and incident process, a monitoring and handover plan, and, where AI is part of the deliverable, a mapping to a recognised risk framework. |
On that last point, the NIST AI Risk Management Framework is intended for voluntary use, so no vendor can be “compliant” with it in any formal sense. NIST released it in January 2023 to help organisations build trustworthiness considerations into the design, development, use and evaluation of AI products, services and systems, which makes it a useful vocabulary for the risk criterion. If the deliverable involves generative AI, NIST also published a Generative AI Profile in July 2024 that is worth asking the vendor about. NIST has said the original framework is being revised, so confirm which version a vendor is referring to. Ask the vendor to say which third-party components its solution relies on and how it would handle a failure in one of them. Treat a clear answer as evidence and a framework logo as nothing.
Have at least two people score independently before comparing. Where two evaluators differ by two points or more on the same criterion, discuss that criterion before averaging, because that gap usually means someone read a clause the other missed.
How to set weights without arguing about them
Weights say what your organisation is prepared to trade. Agree them before the proposals arrive, write them down, and do not change them after you have seen the scores. The default below totals 100, and the two alternative profiles show how the weights might shift for different projects. They are starting points for discussion, not industry benchmarks.
| Criterion | Default | Sensitive-data project | Short proof of concept |
|---|---|---|---|
| 1. Data access and security handling | 15 | 20 | 10 |
| 2. Delivery plan and team | 20 | 15 | 20 |
| 3. Test method and acceptance criteria | 20 | 15 | 25 |
| 4. IP, ownership and exit | 15 | 10 | 10 |
| 5. Cost and commercial terms | 10 | 10 | 15 |
| 6. References and evidence | 10 | 10 | 10 |
| 7. Risk and governance | 10 | 20 | 10 |
| Total | 100 | 100 | 100 |
The calculation is simple: for each criterion, multiply the weight by the score and divide by 4, then add the seven results. A vendor scoring 4 everywhere would reach 100.
Two habits make the weights more honest. First, keep cost modest in the main score, but show it next to the total rather than burying it, so the people approving the budget can see the trade-off. If your procurement policy requires a fixed price weighting, set it first and divide the rest among the other criteria. Second, once the scores are in, re-run the totals with at least one alternative profile. If the ranking holds, the decision is robust. If it flips, you have found a disagreement that belongs in a meeting, not a spreadsheet.
A scored example: two hypothetical data science vendors

This is an illustrative example, not a real procurement. A subscription business wants a churn-prediction model and a monthly report. Vendor A offers the lower fixed price, a confident demo and references from well-known brands. Vendor B costs more, shows less polish, but supplies a test plan. Both passed the four gates after one round of clarification. The default weights apply.
| Criterion | Weight | A score | A points | B score | B points |
|---|---|---|---|---|---|
| 1. Data access and security handling | 15 | 2 | 7.5 | 3 | 11.25 |
| 2. Delivery plan and team | 20 | 2 | 10.0 | 3 | 15.0 |
| 3. Test method and acceptance criteria | 20 | 1 | 5.0 | 3 | 15.0 |
| 4. IP, ownership and exit | 15 | 2 | 7.5 | 3 | 11.25 |
| 5. Cost and commercial terms | 10 | 4 | 10.0 | 2 | 5.0 |
| 6. References and evidence | 10 | 3 | 7.5 | 2 | 5.0 |
| 7. Risk and governance | 10 | 2 | 5.0 | 3 | 7.5 |
| Total | 100 | 52.5 | 70.0 |
The scores follow from evidence, not impressions. Vendor A scored 1 on the test method because it quoted an accuracy figure without saying how it was measured or what it was compared against. It scored 4 on cost because the price was fixed, itemised and tied to milestones. Vendor B scored 3 on the test method because its proposal named a baseline, a held-out validation period and a pass mark. It scored only 2 on cost because the quote was itemised but carried a large optional-extras line that would change the total.
Now stress the result. Suppose leadership decides cost matters more, raising it to 20 and lowering the test method to 10, with the other weights unchanged. Vendor A rises to 60.0 and Vendor B to 67.5. B still leads, though the gap has narrowed from 17.5 points to 7.5. That tells you the ranking is not an artefact of the default weights, while also showing that a strong cost preference would make this a closer call.
One more rule is worth adopting: if any criterion weighted 15 or more scores 1 or lower, hold a clarification round before deciding. Vendor A’s test method score triggers it. A single written answer explaining how the accuracy figure was produced could lift that score, or confirm the concern. Either result is better than discovering it after the contract is signed.
Red flags when evaluating data science vendors
These are practitioner judgements rather than statistics, and none is automatically disqualifying. Each should push the relevant score down until the vendor answers it:
- An accuracy or performance figure with no description of how it was measured, or quoted before the vendor has seen your data.
- Silence on who owns trained models, features and code, or ownership that depends on undefined “standard terms”.
- Data access described as a single export, with nothing on environment, access roles or deletion.
- A timeline that never lists what the vendor needs from your side, such as data extracts, subject-matter experts and sign-offs.
- A price with no assumptions, no change-control process and no list of exclusions.
- References that are logos only, anonymised without explanation, or unavailable for a call.
- Reliance on external models or APIs that the proposal does not mention until you ask.
- A team presented in the pitch that differs from the team named in the SOW.
How to verify a demo
A demo shows what a vendor can present, not what it can deliver on your data. To turn it into evidence, ask for three things. First, run the model or dashboard on a sample of your own data, even a small sanitised extract, instead of a prepared dataset. Second, ask what the model is compared with and how the test data was kept separate from training data. For anything that changes over time, the test period should come after the training period rather than being a random mix. Third, ask to see failure cases, not just the best examples. A vendor that can explain where its approach breaks down is easier to trust than one that cannot find any.
If the vendor cannot show a demo on your data before contract, ask whether a short paid pilot with an agreed pass mark is possible. Score the proposal on what it commits to now, not on what it promises it will prove later.
What to put in a data science vendor contract
The scorecard tells you which proposal looks strongest. The contract is where the items scored 3 become 4s. This list is a starting point for a conversation with your legal team, not legal advice:
- Scope and acceptance criteria that mirror the test plan, including the pass mark and what happens if it is missed.
- Ownership of deliverables, plus a clear licence for any pre-existing vendor tools you depend on.
- Data processing terms covering location, access, subcontractors, retention, deletion and breach notification.
- A handover package: code, documentation, model artefacts and a way to reproduce the environment.
- Named key personnel and a process for approving replacements.
- Change control, milestone-linked payments and clear exclusions.
- Termination and exit assistance, so you are not stranded mid-project.
- Disclosure of third-party components and their licences.
Printable checklist
Copy this into a document and fill it in for each vendor. Score each criterion from 0 to 4 only after you have found the evidence.
Before you open any proposal
- ☐ Weights agreed and recorded (default, or one of the alternative profiles).
- ☐ At least two evaluators named; each will score independently.
- ☐ Pass/fail gates written down.
Gates (pass / fail / clarify)
- ☐ Accepts written data-handling terms.
- ☐ States ownership and licence position in writing.
- ☐ Names the delivery lead and the people doing the work.
- ☐ States how success will be tested.
Scores (0–4, with the evidence you found)
- ☐ Data access and security handling: ____ (weight ____)
- ☐ Delivery plan and team: ____ (weight ____)
- ☐ Test method and acceptance criteria: ____ (weight ____)
- ☐ IP, ownership and exit: ____ (weight ____)
- ☐ Cost and commercial terms: ____ (weight ____)
- ☐ References and evidence: ____ (weight ____)
- ☐ Risk and governance: ____ (weight ____)

Before you decide
- ☐ Evaluators compared scores and discussed any gap of two points or more.
- ☐ Any criterion weighted 15 or more that scored 1 or lower has had a clarification round.
- ☐ Totals re-run with at least one alternative weighting.
- ☐ Demo claims checked on a sample of your own data, or a pilot agreed.
- ☐ Contract draft reflects every item you scored 3.
Data science vendor evaluation: frequently asked questions
What should a data science vendor evaluation checklist include?
It should include pass/fail gates for data handling, ownership, a named delivery lead and a stated success test, followed by weighted scoring of data access, delivery, test method, ownership, cost, references and risk. Every score should rest on evidence you can check.
How much weight should cost get when scoring data science vendors?
In the default weighting here, cost carries 10 of 100 points, and the short proof-of-concept profile raises it to 15. Show cost beside the total so budget holders can see the trade-off, and re-run the totals with a higher cost weight to test whether the ranking holds.
How do you compare data science proposals fairly?
Agree the weights before opening any proposal, have at least two evaluators score independently, and discuss any criterion where scores differ by two points or more. Then re-run the totals with an alternative weighting.
Next step
Run the checklist on the proposals you already have, starting with the gates. If two vendors come out close, spend your time on the criteria where their scores differ most and ask for written answers rather than another meeting. If you would like a proposal written so that each criterion above can be checked line by line, with assumptions, a test plan and ownership terms stated, you can discuss a scoped project with DataColab, which offers custom AI and ML software services and typically begins with a defined piece of work that may continue afterwards.