Specialist · $149
Code and Reasoning Data
Code and mathematical reasoning data commands some of the highest rates in AI training because verification requires real expertise. Built for people with a computer science, engineering, or mathematics background.
Join the enrolment list
Enrolment is not open yet. Leave your details and we will email you when it opens, at $149.
What to expect
- Around 45 verification exercises on generated code and derivations, each with the defect identified
- Language agnostic reasoning, with worked examples in Python and a little SQL
- Written verification notes that are graded like a specialist project would grade them
- The longest of the courses, because verification cannot be skimmed
What is included
- 45 verification exercises with annotated defects
- Two graded assignments: a test suite and three verification notes
- A test case design reference for adversarial inputs
- A verification note template used on specialist projects
- A partial credit rubric for grading long derivations
Before you start: Able to read code in at least one language and reason about correctness. Comfortable with mathematics to about first year undergraduate level.
Who this is for
- People with a computer science, engineering, mathematics, or physics background
- Working developers who want paid evaluation work that uses the skill they already have
- Graduate students looking for technical work that pays better than general annotation
Skip it if
- Anyone without a technical background, since the exercises assume you can read code and follow a proof
- People looking to learn programming, which this does not teach
What you will be able to do
- Evaluate code written by a model for correctness, not just plausibility
- Grade reasoning written out step by step and locate the first wrong move
- Write test cases and repair prompts that expose real defects
- Document verification work to the standard specialist projects require
- Judge partial credit consistently on a long derivation
- Pass the technical screens used on code and mathematics projects
Syllabus
6 modules, 26 lessons, about 11 hours of work in total.
- 01
Verifying generated code
1 hr 30 minCorrectness, complexity, and silent failure, beyond whether it runs.
- Reading generated code for intent before reading it for bugs
- Correctness, complexity, and the silent failure that passes the happy path
- Code that merely looks plausible, and the tells that give it away
- Exercise set: 15 samples with the defect identified
- 02
Grading reasoning one step at a time
2 hrLocating the first invalid step and scoring partial credit consistently.
- Locating the first invalid step rather than the first surprising one
- Scoring partial credit consistently across a long chain
- Right answer, wrong reasoning: how to score it and why it matters
- Reasoning that skips a step versus reasoning that assumes the conclusion
- Exercise set: 15 graded derivations with a model score
- 03
Mathematical verification
2 hrChecking derivations and proofs, and handling answers that are right for the wrong reason.
- Checking a derivation without redoing it from scratch
- Proof checking: structure, gaps, and unjustified leaps
- Numerical answers, units, and the errors that survive a sanity check
- Symbolic manipulation errors models make repeatedly
- Exercise set: 15 derivations with annotated faults
- 04
Writing adversarial test cases
1 hr 55 minConstructing inputs that separate a working solution from a lucky one.
- Designing an input that separates a working solution from a lucky one
- Boundaries, empty inputs, and the cases generated code forgets
- Performance cases that expose the wrong algorithm
- Assignment: a test suite for three generated solutions
- 05
Specialist documentation standards
1 hr 50 minEvidence and reproducibility expectations on the best paid projects.
- What a specialist reviewer needs in order to trust your verdict
- Writing a verification note another engineer can reproduce
- Citing evidence: the failing input, the expected result, and the actual one
- Assignment: three verification notes graded against the template
- 06
Getting onto specialist projects
1 hr 45 minTechnical screens, rate expectations, and proving domain depth.
- How technical screens differ from general annotation assessments
- Presenting a technical background so it survives a screening pass
- Rate expectations by domain, and where the premium actually is
- Keeping a defensible record on projects that audit heavily
Roles this prepares you for
Free reading first
These guides are free and cover some of the same ground. Read them before paying for anything.
Questions about this course
How much programming do I need?
+
Enough to read unfamiliar code in at least one language and reason about whether it is correct. You do not need to write much code, but you do need to spot a wrong algorithm.
Which language are the examples in?
+
Mostly Python, with a little SQL, because that is what appears most often on these projects. The skills are language agnostic and the reasoning transfers.
Does a technical background really pay more here?
+
It is where the widest rate gap sits, because verification cannot be outsourced to someone who does not understand the domain. The salary guide has the current ranges.
Before you buy
This is training you work through at your own pace, written by DataAnnotationJobs.org. It does not guarantee a job, an assessment pass, or any level of earnings, and it is not affiliated with or endorsed by any employer listed on this site. See terms and refunds.