AI Training Data Labeling: Quality Standards, Tools and When to Outsource
Key takeaways
- Consistency beats cheap tagging. Quality comes from a taxonomy tied to a real decision, written guidelines that settle borderline cases, and a measured review loop.
- Label errors in your test set cap how accurately you can measure a model. A model can look several points worse than it is, and you cannot rank close candidates.
- Raw percent agreement misleads when one label is rare. Use a chance-corrected statistic and set the threshold from a pilot.
- Outsource only after the basics exist. You need written guidelines, a gold set and a way to audit the output first.
AI training data labeling is rarely hard because of the tool or the hourly rate. The hard part is getting several people to make the same judgement on ambiguous examples, and being able to prove it. Good labels are also the foundation for any plan to create a machine learning model fast, because a quick model trained on inconsistent labels only produces inconsistent results quickly.
This guide covers label types, taxonomy, guidelines, gold sets, quality assurance, privacy, tooling and outsourcing. It includes three working artifacts: an annotation guideline, a label audit sampling sheet and a disagreement rubric. Every example is a hypothetical scenario you can adapt.
Which label type fits your data labeling project?
The kind of label you need determines how agreement is judged, how long each item takes and where the effort goes. Decide this before writing any guidelines.
| Label type | Example | How agreement is usually judged |
|---|---|---|
| Single-label classification | One intent per support message | Exact match, plus a chance-corrected statistic such as Cohen’s kappa |
| Multi-label classification | Several topics per document | Per-label agreement, or set overlap (Jaccard similarity) |
| Span labelling (named entities, PII) | Marking customer names in text | Overlap of the marked spans plus label match |
| Boxes and masks | Defects on a product image | Region overlap (intersection over union) plus label match |
| Ranking or preference | Which of two answers is better | Pairwise agreement; expect real disagreement on subjective items |
| Free-text or rubric scoring | Score an answer 1–5 against criteria | Calibration sessions and rubric-level agreement |
Agreement on a span or a box needs an overlap rule, and you have to choose that rule on purpose.
How to design a data labeling taxonomy
A taxonomy is the list of labels and the rules for choosing between them. Work backwards from what the model’s output will trigger. If a predicted “cancellation” routes a message to a retention team, the cancellation label must be defined tightly enough that the team gets what it expects.
- Start smaller than you think. Every extra label multiplies the borderline cases. Add a label when a distinction changes an action, not because it is interesting.
- Include an explicit “other / unclear” label. Without one, annotators force-fit ambiguous items into the nearest label and hide the taxonomy’s gaps.
- Make labels mutually exclusive, or say they are not. If an item can legitimately carry two labels, make the task multi-label rather than relying on annotators to guess a tie-break.
- Merge labels that people cannot separate. If two labels never reach acceptable agreement after the guideline is clarified, the distinction is probably not recoverable from the data. Merge them, or collect more context.
- Test the taxonomy on a pilot. Use real items, labelled independently by two or three people, before you commit to a large batch. Most taxonomy defects show up in the first few dozen items.
Annotation guidelines that settle borderline cases
Guidelines are where consistency is won. Each label needs a definition, a “do not use when” boundary, and examples of the cases people actually argue about. Below is a compact, adaptable guideline for an illustrative example: assigning one primary intent to inbound customer messages for a hypothetical B2B invoicing product.
Example guideline: primary intent of a support message
| Label | Use when | Do not use when |
|---|---|---|
| billing_question | The customer asks about a charge, invoice, refund or failed payment. | The customer wants to end the subscription (use cancellation). |
| technical_fault | A feature that should work is failing or behaving unexpectedly. | The customer cannot sign in at all (use account_access). |
| account_access | The customer cannot log in, reset a password or reach their account. | They can log in but one feature is broken. |
| cancellation | The customer explicitly asks to cancel, downgrade or close the account. | They only complain or threaten to leave without asking for cancellation. |
| feature_request | The customer asks for something the product does not currently do. | The feature exists but is failing (use technical_fault). |
| other_unclear | The message has no actionable request, or you cannot tell what is wanted. | You are unsure between two labels; apply the rules below first. |
Tie-break rules

- Label the thing the customer needs the support team to do first. Ask: “What must happen before this message can be considered resolved?”
- If one problem blocks the customer from using the product (for example, they cannot log in), label the blocking problem and tick multi_intent.
- An explicit request outranks a complaint. “Cancel my plan” is cancellation, even if the reason given is a bug.
- If rules 1–3 still leave two plausible labels, choose other_unclear and add a one-line note. Do not guess.
Worked examples
| Message (hypothetical) | Label | Reason |
|---|---|---|
| “I was charged twice for March and now I can’t log in to download the invoice.” | account_access, multi_intent | Login blocks the customer from everything else (rule 2). |
| “Please cancel my plan, the sync keeps failing anyway.” | cancellation | Explicit request beats the complaint (rule 3). |
| “My export has said ‘processing’ for two days.” | technical_fault | An existing feature is failing. |
| “Any plans to support multi-currency invoices?” | feature_request | Asks for something the product does not do. |
| “My card was declined but I wasn’t charged.” | billing_question | Failed payment is within billing. |
| “Thanks!” | other_unclear | No actionable request. |
To adapt this, replace the labels with your own, keep the “do not use when” column, and add every disputed example from your pilot to the worked examples. Version the document and record which version each batch was labelled under. That way you can tell later whether a change in agreement came from the data or from a rule change.
How to build a gold set before you scale
A gold set is a smaller collection of items whose labels have been double-checked, used to evaluate models and to test annotators. To build one:
- Have at least two qualified people label each item independently.
- Let a named domain owner adjudicate every disagreement using the written guidelines.
- Keep the gold set out of training data, and record how each label was reached.
- For genuinely ambiguous items, either leave them out or store an accepted set of labels, so a model is not penalised for choosing a defensible answer.
Why label errors distort model evaluation
Evaluation is where label quality shows up as a business problem. If your test labels contain errors, measured accuracy drifts away from true accuracy. Here is a simple illustrative calculation. Assume a binary task, label errors that occur at random, and a model with true accuracy p. When a model and a label are both wrong on a binary task, they agree by coincidence, so measured accuracy is p×(1−e) + (1−p)×e, where e is the label error rate.
| Label error rate in test set | Measured accuracy of a 95%-accurate model | Measured accuracy of a 90%-accurate model |
|---|---|---|
| 0% | 95.0% | 90.0% |
| 2% | 93.2% | 88.4% |
| 5% | 90.5% | 86.0% |
| 10% | 86.0% | 82.0% |
Two practical lessons follow.
- Noisy test labels compress differences between models. A real five-point gap shrinks to 4.5 points at 5% label error, and small improvements disappear into noise.
- Real label errors are rarely random. If annotators share a systematic misreading of a rule, a model that learned the same bias can look better than it is. That is why adjudication rules and the audit loop below matter more than a headline accuracy figure.
Measuring annotator agreement and auditing label quality
Chance-corrected agreement (Cohen’s kappa)
Percent agreement is easy to calculate and easy to misread. Statistics such as Cohen’s kappa and Krippendorff’s alpha correct for agreement that would happen by chance. Cohen’s kappa compares observed agreement po with the agreement expected by chance pe: κ = (po − pe) / (1 − pe), as defined in the scikit-learn documentation.
Here are two hypothetical batches of 100 items, each labelled by two annotators on a yes/no question (“is this a billing message?”).
| Batch | Both “yes” | A yes, B no | A no, B yes | Both “no” | Raw agreement | Cohen’s kappa |
|---|---|---|---|---|---|---|
| Balanced | 40 | 8 | 7 | 45 | 85% | 0.70 |
| Rare label | 2 | 2 | 3 | 93 | 95% | 0.42 |

The rare-label batch has higher raw agreement and much lower kappa. Most of that 95% comes from both annotators saying “no” on easy negatives, while they barely agree on the positives you care about. Check agreement per label, not just overall.
The calculation takes a few lines of Python. Replace the lists with your two annotators’ labels for the same items, in the same order:
from sklearn.metrics import cohen_kappa_score
annotator_a = ["billing", "other", "billing", "other"] # replace with real labels
annotator_b = ["billing", "other", "other", "other"]
kappa = cohen_kappa_score(annotator_a, annotator_b)
print(round(kappa, 3))
For more than two annotators or missing labels, use a statistic designed for that, such as Krippendorff’s alpha. No single kappa value is universally “good enough.” Acceptable agreement depends on how subjective the task is and what an error costs. Set your threshold from the pilot, write it down, and revisit it when the taxonomy changes.
Disagreement rubric
Every disagreement is information, but only if you classify why it happened. Use this rubric when an adjudicator reviews a disputed item.
| Code | What it looks like | Action |
|---|---|---|
| D1 – Annotator slip | One label is clearly wrong under the current guideline. | Correct the label; feed back to the annotator; no guideline change. |
| D2 – Guideline gap | The guideline says nothing about this case. | Add a rule and a worked example; relabel similar items. |
| D3 – Guideline conflict | Two rules apply and point to different labels. | Add a precedence rule; version the guideline. |
| D4 – Genuinely ambiguous item | Reasonable experts differ even with a clear guideline. | Exclude from the gold set or store an accepted label set; consider whether the model should abstain. |
| D5 – Taxonomy defect | Two labels overlap, or no label fits. | Merge, split or add a label; re-run the pilot. |
Tally the codes per batch.
- Many D1s: a training or workload problem.
- Many D2s and D3s: the guideline is incomplete.
- Many D5s: stop labelling and fix the taxonomy, because more volume will only add more inconsistent data.
Label audit sampling sheet
Agreement between annotators does not prove the labels are right. Annotators can agree and still share a misunderstanding. An audit has a separate reviewer re-check a sample against the guideline. This illustrative plan assumes a weekly batch of 4,000 labelled messages and five annotators.
| Stratum | Why sample it | Sample size | What the reviewer records |
|---|---|---|---|
| Random across all labels | Estimate the overall error rate | 120 | Correct / wrong / guideline gap |
| Each rare label (for example cancellation, feature_request) | Rare labels vanish in a random sample | 30 per label | Correct / wrong; confusion with which label |
| Items adjudicated after disagreement | Check adjudication is consistent | 40 | Rubric code D1–D5 |
| Newest annotator’s first batches | Catch misunderstandings early | 40 | Correct / wrong; error pattern |
| Items marked multi_intent | Test the tie-break rules | 30 | Was the blocking problem chosen? |
| Items labelled other_unclear | Find taxonomy gaps | 30 | Real gap, or a missed existing label? |
That is roughly 320 reviews, or about 8% of the batch. Items that fall into more than one stratum should be counted once. Scale the plan to your reviewer capacity, but do not shrink the random sample casually. With 120 reviewed items and 3 errors found (2.5%), a rough 95% interval from the normal approximation is about ±2.8 percentage points, so the true error rate could plausibly be near 5%. Showing that error is under 3% takes a much larger sample. Pass rules should include the target error rate, what happens on a fail (relabel the batch, retrain annotators or fix the guideline), and who signs off.
When to audit edge cases
Audit edge cases earlier and more often than your random sample suggests:
- during the pilot
- for the first batches of any new annotator
- after any guideline change
- whenever a new data source is added
- whenever the weekly tally shows a rising share of D2, D3 or D5
Keep an edge-case log with the item, the decision, the reasoning and the guideline version, and fold stable decisions back into the worked examples.
Privacy, consent and data rights in data labeling
Labelling means people read your data. Before an item is shown to an annotator, decide whether you have the right to use it for this purpose. If it contains personal data, data-protection law such as the UK or EU GDPR may apply, and legal review is a sensible step, not an afterthought. A clear data governance strategy for compliance and trust should settle these questions before any data reaches annotators. Then reduce exposure:
- Minimise and mask. Remove or redact identifiers that annotators do not need. Show the minimum context a label requires.
- Control access. Use named accounts, role-based permissions and an audit trail of who saw and changed what.
- Check where the data goes. If a vendor or tool processes the data, confirm where it is stored, who can access it and what the contract says about retention, deletion and reuse for the vendor’s own models.
- Protect annotators too. Content that is distressing or sensitive needs warnings, limits and a way to opt out.
The NIST AI Risk Management Framework is a useful vocabulary for documenting data provenance, quality checks and accountability around a dataset. NIST describes it as voluntary and intended to help organisations build trustworthiness considerations into the design, development, use and evaluation of AI systems. It was released on 26 January 2023, and NIST has since published a Generative AI Profile (July 2024). NIST also notes that AI RMF 1.0 is being revised, so check the page for the current version. It is not a legal compliance checklist.
How to choose a data labeling tool
Pick the tool after you know your label types, privacy constraints and review process. These questions separate tools in practice:
- Does it support your label type, including overlap rules for spans or regions?
- Can you assign overlap, so the same item goes to several annotators, and see agreement per label?
- Is there a review or adjudication queue, and can you keep ground-truth labels separate?
- Can annotators see the guideline inside the interface?
- Does it keep an audit trail and export labels with annotator and version metadata?
- Can it show model pre-labels, and can you measure whether pre-labels bias annotators?
- Where is data hosted, and what access controls exist?
Check the edition you are buying. The Label Studio documentation compares its editions. It lists agreement metrics as unavailable in the open-source Community Edition, limited in Starter Cloud and available in Enterprise. Overlap configuration, reviewer assignment and ground-truth marking are listed for the paid editions rather than Community. Importing pre-annotated data and connecting a custom ML backend are available in all editions. A team on the free edition may therefore need to export labels and compute agreement itself, as in the Python example above. Features change, so confirm current capabilities with any vendor before you commit.
In-house, vendor, crowd or model-assisted labeling: when to outsource
| Option | Fits when | Main risk |
|---|---|---|
| In-house subject experts | Labels need deep domain judgement or data is highly sensitive; volume is modest. | Expert time is expensive and hard to scale. |
| Managed specialist vendor | You need volume with trained, supervised annotators and a service-level agreement. | Guideline drift; less visibility into who labels what. |
| Crowd marketplace | Tasks are simple, well defined and the data is not sensitive. | Variable quality; needs strong gold-set checks. |
| Model-assisted labelling with human review | A first-pass model is decent and reviewers can correct quickly. | Reviewers may anchor on the model’s suggestion and miss errors. |
Outsource when volume or expertise outgrows your team, but keep three things in-house: the taxonomy, the gold set and the acceptance decision. When you brief a vendor, supply the guideline, the gold set and your audit plan. Ask how they measure agreement, how they handle disagreements and which of their reviewers are qualified for your domain. If you need engineering support around the labelling pipeline, custom AI and ML software services can cover tooling, pre-labelling models and integration.
Be open about human review in your own dataset documentation as well. Record who labelled the data, whether a model pre-labelled it, how disputes were resolved and which guideline version applied. Anyone relying on the dataset, including your future self, needs that context to interpret model results.
Frequently asked questions
What is AI training data labeling?
It is the process of attaching labels, such as categories, spans, boxes or scores, to raw data so a model can learn from it and be evaluated against it.
How do I measure data labeling quality?
Use a chance-corrected agreement statistic such as Cohen’s kappa, check it per label rather than only overall, and add a separate audit where a reviewer re-checks a sample against the guideline.
Why is percent agreement not enough?
When one label is rare, easy negatives inflate raw agreement. In the example above, a batch with 95% raw agreement has a kappa of only 0.42.
When should I outsource data labeling?
When volume or specialist expertise outgrows your team, and only after you have written guidelines, a gold set and an audit plan.
A sensible next step
Before you commission a large batch, label 50 to 100 real items with two or three people working independently. Count the disagreements, classify them with the rubric above, and fix the guideline. If the pilot cannot reach acceptable agreement on a label, no amount of volume will rescue it.