Data annotation job
AI Developer Trace Task Auditor
Mercor · Posted 2 months ago
- AI evaluation
- Coding
About this role
Evaluate the quality and correctness of AI-assisted software-development traces used to train and evaluate a frontier AI lab's models. You'll assess end-to-end coding sessions produced with AI-assisted developer tools, judging correctness, workflow soundness, and reasoning, and provide clear, rubric-based written feedback.
Basic Qualifications
- 3+ years professional software development
- Hands-on experience with AI-assisted coding tools and agentic / spec-driven workflows (Cursor, GitHub Copilot, Claude Code, or similar)
- Strong code-reading and debugging skills across full-stack or backend systems
- Ability to evaluate multi-step coding trajectories for correctness and best practice
Preferred Qualifications
- Experience with Kiro or Amazon CodeCatalyst
- Prior work evaluating or grading AI-generated code
- Contributions to developer tooling
This listing is aggregated by DataAnnotationJobs.org. Applications are processed by Mercor, not by this site.
Mercor is an AI-native talent marketplace connecting vetted domain experts such as doctors, lawyers, and engineers directly with frontier AI labs for RLHF, evaluation, and specialized training data at massive scale. See all Mercor jobs.
Apply for this role
Your details first, then a short screening test. Both are read as part of your application and go to Mercor, and we will tell you when similar roles appear.