Reference

Data annotation glossary

Terms you will see on this board and in employer assessments. For hiring context, pair a definition with open jobs or the RLHF explainer.

Data annotation

Paid human work that turns raw text, images, audio, or video into labeled examples a model can learn from. In 2026 it often includes ranking chatbot answers and writing better responses, not only drawing boxes on photos.

Data labeling

A narrower phrase for tagging items to a schema: categories, spans, transcripts, or regions. Teams still say labeling for classic tasks; RLHF and evaluation sit on top of that idea.

RLHF

Reinforcement learning from human feedback. People compare model outputs, pick a winner, and often write a short justification. Labs use those preferences to steer assistants toward helpful, honest, harmless behavior.

SFT (supervised fine-tuning)

Training a model on pairs of prompts and responses written or edited by humans. If RLHF is “which answer is better?”, SFT is “here is a good answer.” Many AI trainer jobs mix both.

Preference data

Pairs or rankings of model answers with a human choice. This is the raw material for RLHF. Thin or inconsistent preferences make the reward model noisy, which is why QA is strict.

AI trainer

Someone who teaches models with examples and judgments rather than only applying a fixed label set. Overlaps with RLHF rater, expert annotator, and writing-heavy LLM work.

LLM evaluator

A rater who scores model answers for accuracy, style, safety, or rubric fit. Evaluation can be a production job (ongoing QA) or a project to benchmark a new model.

Gold standard

An expert-agreed “correct” label or response used to measure other annotators. If your work disagrees with gold too often, you fail hidden QA even when the task felt subjective.

Inter-annotator agreement

How often two people give the same label on the same item. Low agreement usually means a fuzzy rubric, not that “everyone is right.” Platforms rewrite instructions when IAA collapses.

Bounding box

A rectangle drawn around an object in an image or video frame. Common in vision and autonomous-vehicle pipelines. Tightness and occlusion rules are usually spelled out in the project guide.

NER (named-entity recognition)

Marking names of people, places, organizations, dates, and similar spans in text. Guidelines decide edge cases (nicknames, nested entities, product names).

Transcription

Turning speech into text, sometimes with timestamps, speaker labels, or verbatim vs cleaned conventions. Audio annotation still shows up alongside LLM work.

Trust and safety

Reviewing content or model output for policy violations: hate, self-harm, scams, adult content, or dangerous advice. Emotionally heavy; read the wellness and escalation rules before you take the queue.

Red teaming

Trying to make a model fail on purpose (jailbreaks, biased answers, leaking private data) so the lab can patch it. Closer to security and policy work than to simple labeling.

QA (quality assurance)

Sampling finished tasks against a gold set or a senior reviewer. High-volume platforms fire or pause workers whose QA score drops. Speed that tanks QA is not a win.