What is data annotation? Tasks, quality, and why the jobs exist

What is data annotation? A plain definition

Take an input a computer cannot reliably score on its own: a photo, a paragraph, a clip, a pair of chatbot answers. A trained person attaches structure to it. That structure might be a class name, a box, a transcript, a preference, or a rewritten gold answer. The pile of those decisions becomes a dataset or an evaluation set. That process is data annotation. The person is a data annotator. When a company pays for it, you have a data annotator job.

Here is the concrete version. A retailer wants software that reads incoming messages and routes them. Someone hands you a message that says the delivery arrived on time but the box was crushed and could somebody call back today. You mark it as a complaint rather than a question, tag the product mention, note the urgency, and move on. Do that ten thousand times with a consistent definition of complaint and you have a training set. Hold back a slice of it and you have an evaluation set that tells the team whether the model is getting better. Neither exists without the human pass.

The data annotation meaning that matters in practice is narrower than the dictionary one: you are not describing the data, you are applying somebody else’s definition to it. The definition lives in a guideline document. A guideline author decided, months before you arrived, whether a delivery robot counts as a vehicle, whether a nickname counts as a person entity, and whether a refusal is the correct answer to a borderline medical question. Your value is consistency with that decision, not the quality of your personal taste.

Related phrases are not rival industries. Labeling, tagging, rating, transcription, and RLHF are task families inside the same job. We keep data labeling for closed schemas, data annotation for the broad activity, and preference data for the ranking work that trains assistants. A single project frequently mixes two of them in one queue.

The annotation pipeline and who does what

Most explanations of data annotation stop at the annotator, which makes the work look lonelier and simpler than it is. A real project has several roles, and knowing them explains why your instructions feel second-hand and why nobody answers your question quickly.

The structural point is that the company running your queue is usually not the lab that wanted the data. Vendors sit between a frontier lab and you. That explains the non-disclosure agreements, the fact that you often will not be told whose model you are rating, and why a project can vanish the week a lab changes priorities. It also explains the mid-project addendum: the lab looked at a sample, disliked something, and a new page appeared in the rubric on a Tuesday.

The useful career implication is that the ladder is visible from the inside. Annotators become reviewers, reviewers become adjudicators, and people who write clearly end up drafting guidelines instead of following them. That progression, and what each rung tends to list, is mapped in the data annotator salary guide.

Types of data annotation, by modality

There are more task names than there are genuinely different jobs. The honest taxonomy sorts by modality first, then by whether you are picking from a closed set or producing free text.

Classification and spans

Pick a category, or highlight the exact words that carry a meaning. Named entity recognition is the textbook case: mark people, organisations, and locations. It sounds trivial until the sentence contains a football club named after a city, at which point the guideline has to tell you whether that is a place or an organisation, and you have to remember which.

Spatial and temporal work

Bounding boxes, polygons, keypoints, and video tracks. Autonomous driving made this category famous, and occlusion rules fill half the document: what to do when a pedestrian is two-thirds behind a van, whether a cyclist is also a person, how to handle an object clipped by the frame edge. The separating skill is spatial patience at pixel level for hours without drifting.

Transcription and audio

Transcription, sometimes with speaker turns and timestamps. Verbatim versus cleaned is a running argument in every guideline: whether to keep the false starts, the “um,” the repeated word, the half-finished sentence. Neither answer is wrong in general, and only one of them is right on your project.

Generation and preference

Write the answer you wish the model had given, or choose between two and explain the choice. This is where data annotation collided with assistant products, and it is the fastest-growing family on the board. The concept is covered in what is RLHF, the practice in the prompt and response writing course, and the supervised fine-tuning entry explains why a hand-written answer is worth more than a rating.

Policy and safety

Trust and safety queues and red teaming. Still annotation, because you are attaching a human judgment to an artifact, but with a policy document instead of a schema and real emotional weight attached to the queue.

One ambiguous item, worked through

Definitions get abstract fast, so here is a single item of the kind that consumes half of a new annotator’s first week. The project labels objects in dashcam footage. The frame shows a parked delivery robot on the pavement, roughly the size of a suitcase, with a company logo on the lid. The label set contains vehicle, pedestrian, cyclist, and static object.

Every instinct is defensible. It has wheels and moves under its own power, so vehicle. It is not carrying a person and is not on the road, so static object. It behaves like a pedestrian in traffic modelling terms, so pedestrian. Three reasonable annotators will pick three different answers, and the dataset now contains noise that a model will faithfully learn.

A good guideline resolves this with a decision rule rather than a vibe: something like classify by whether the object occupies the roadway and moves under power, with a named exception list that includes delivery robots as static objects when stationary. Your job is to find that rule in a long document, apply it, and note the item in the project channel if the rule is silent. What you must not do is guess twice, because inconsistency from a single annotator is more damaging than a consistent wrong call, which at least can be corrected in bulk.

The same pattern appears in language work with different props. Two model answers, one warmer and more detailed, one shorter and refusing a restricted request. The rubric ranks safety above helpfulness, so the colder answer wins even though it reads worse. Learning to feel the difference between what you prefer and what the document requires is the actual skill this industry hires for, and it is the thing assessments test. The tactics are in how to pass annotation assessments.

How annotation quality is measured

If two annotators disagree, the dataset is noise, and noise is what a lab refuses to pay for. So quality is not a compliance layer bolted onto annotation. It is the product. Four mechanisms do most of the work.

• Gold sets. A gold standard is a batch of items with a known correct answer, seeded invisibly into production. You cannot tell which items they are, which is the point. Your hit rate on them is the closest thing to a performance review you will get.
• Inter-annotator agreement. Agreement scores compare you against everyone else on the same items. Sitting in the tail of that distribution is how people quietly stop receiving work, often without an explicit rejection.
• QA sampling. A reviewer pulls a percentage of your output and corrects it. QA feedback usually arrives as a score change plus a terse note rather than a conversation.
• Adjudication. Contested items go to a senior rater whose call becomes the new rule. If you see the same ambiguity three times, raising it is more valuable than answering it.

This is why assessments feel petty. They are not testing intelligence or writing flair. They are measuring whether you share the same imaginary correct as the author of the document, and whether you will keep sharing it after the novelty dies. Speed without agreement is how a vendor loses a lab contract, so the vendor pauses you long before the lab notices.

The practical takeaway for anyone hoping to get paid: your quality score in the first two weeks matters more than anything on your resume. Read the feedback, redo flagged items the way the reviewer asked, and ask one clarifying question in the project channel rather than guessing twice. The free AI data annotation foundations course walks through reading a rubric the way a reviewer reads it.

Annotation, labeling, collection, curation

Four terms get used loosely in job ads and vendor marketing, and the differences matter mainly because they predict what your day looks like. The data labeling versus data annotation question in particular has no clean industry-wide answer, so treat the table as a working convention rather than a standard.

There is one more distinction worth knowing because it confuses searchers constantly. DataAnnotation.tech is a specific contractor platform, not a generic phrase, and this site is an independent job board that lists openings from many companies. The untangling is in DataAnnotation careers versus this board. For the hiring-side view of the AI-branded variants of all these terms, read AI annotation jobs.

Why the jobs did not vanish

The obvious question about data annotation in 2026 is why humans are still doing it. Models did get better at cheap labels. Model-assisted pre-labeling now handles the unambiguous majority of many datasets: a model proposes the box or the category and a person confirms or corrects it. Synthetic data covers a slice of what used to be hand-written examples. Both of those genuinely removed work.

What they removed was the bottom of the stack. The remaining human queue is the part a model cannot settle without circular reasoning. Preference data needs a human because the whole point is to capture what people prefer. Evaluation sets need humans because a model grading itself tells you nothing about whether it is right. Expert domains need credentials the labor pool is short of. Safety and policy calls need someone accountable. And every pre-labeled dataset still needs an adjudicator for the cases where the model was confidently wrong.

So the shape of the work moved rather than shrank: less box drawing as a share of the total, more RLHF, more LLM evaluation, more writing, more specialists. If your mental model of annotation comes from a 2018 course about image datasets, read a current job card before you assume you already know the queue. The current market is described in data annotation jobs in 2026, and the geography of it in data annotation jobs worldwide.

Be careful with market-size claims in either direction. Numbers on vendor slides are estimates, and nobody publishing them is neutral. The reliable signal is duller: listings keep appearing, assessments keep being run, and the mix on those listings keeps tilting toward judgment.

Working conditions and ethics, honestly

Any honest explanation of data annotation has to include the conditions, because the same properties that make the work accessible also make it precarious. None of this is a reason to avoid the field. It is a reason to go in with your eyes open.

• Contractor status is the norm. No notice period, no paid idle time, your own taxes, and hours that expand or vanish with a project. Treat a listed hourly rate as the ceiling of a good week rather than the base of a predictable month.
• Feedback is one-directional. You will rarely speak to the reviewer grading you, and a score can drop without an explanation you find satisfying.
• Safety queues carry real exposure. Policy and trust and safety work means sustained contact with material you cannot unsee. Read the wellness rules and break policy before accepting, and do not treat these queues as an easy entry point because the floor rate is lower.
• Non-disclosure is standard. You often will not know whose model you are improving, which makes it hard to judge what your labour is being used for.
• Attribution is thin. Annotated data is the foundation of systems whose credit goes elsewhere. That gap is a legitimate criticism of the industry, not a personal failing.

The counterweight is that the work is a genuine door into AI for people without a degree in it, it is remote in most cases, and the skill it builds, reading a specification and applying it consistently, transfers to evaluation design, operations, and quality roles. The ladder is real if you treat the first project as a credential rather than a destination.

From definition to paid work

If the concept is clear and you want a seat, the route is short and unglamorous. Nothing below is clever, and almost every stalled application skipped one of the five.

1. Learn to read a guideline like a reviewer. Before you apply anywhere, practise finding the rule that decides an ambiguous item instead of answering from instinct. This single habit is what assessments measure, and it is the difference between a pass and a polite rejection.
2. Pick one task family to be good at. Choose labeling, response writing, preference rating, or evaluation. Each has a different rubric style and a different tool. Applying to all four at once usually produces four rushed assessments rather than one qualification.
3. Check eligibility on live listings before you invest time. Open the jobs feed and read the location line first. Remote global, remote US-only, and on-site are three different markets, and eligibility rejects more applicants than skill does.
4. Pass one assessment properly. Read the whole rubric before item one, assume hidden gold items are mixed in, and write justifications that name the rule you applied. Do not paste chatbot prose, because screens now flag fluent generic paragraphs.
5. Protect your quality score, then add a second vendor. Your first two weeks of samples decide how much volume you see. Act on reviewer feedback rather than arguing with it, then qualify with a second vendor while your first queue is healthy so a paused project does not become a zero week.

For the on-ramp in detail, read how to become a data annotator and the rest of the guides library. To gauge where you stand before spending an attempt on a real test, run the skills assessment and pick a track from the course list. If you want to know who runs the large programs, the companies index covers Scale AI, Surge AI, and the rest, and the Scale AI versus Surge AI comparison is the fastest way to choose between the two biggest names. Beginners should start at beginner and entry-level roles.

FAQ: data annotation meaning, types, and examples

One-line definitions of every term used above live in the glossary. Live openings are on the jobs feed, and the remote-only view is remote data annotation jobs.