Data annotation job
Legal Rubric & Ground Truth Auditor (Licensed Attorney)
Mercor · Posted 2 days ago
- AI evaluation
- Healthcare
- Legal
- Writing
About this role
Mercor is seeking experienced licensed attorneys to audit rubrics and ground-truth answers for a legal research dataset used to evaluate advanced AI systems. You will review existing legal research questions, verify the accuracy of model answers, and refine grading rubrics so the dataset reflects current law and sound legal reasoning. This is a short-term, fully remote engagement.
Responsibilities
- Audit grading rubrics for accuracy, completeness, consistency, and clarity.
- Verify ground-truth answers against controlling statutes, regulations, and case law.
- Identify legal errors, outdated authority, missing issues, and ambiguous grading criteria.
- Rewrite or correct ground-truth answers and rubric items where needed.
- Provide concise, structured written feedback on each item reviewed.
Practice Areas
The dataset spans eight practice areas. Experience in one or more is required, and breadth across several is a plus:
- Administrative / Regulatory
- Business & Commercial
- Civil Litigation
- Constitutional / Civil Rights
- Criminal
- Family
- Health
- Immigration
Required Qualifications
- Juris Doctor (JD) and active U.S. bar admission in good standing.
- 5+ years of legal practice.
- Substantive experience in at least one of the practice areas above.
- Strong legal research, analysis, and writing skills.
- High attention to detail and the ability to apply consistent judgment.
Preferred Qualifications
- Experience across multiple listed practice areas.
- Judicial clerkship, appellate, government, or law firm research experience.
- Prior experience reviewing or grading legal work product, or with AI / legal-tech evaluation projects.
Why Join?
- Apply your legal expertise to shape how AI systems are evaluated for legal research.
- Collaborate with a leading AI research organisation.
This listing is aggregated by DataAnnotationJobs.org. Applications are processed by Mercor, not by this site.
Mercor is an AI-native talent marketplace connecting vetted domain experts such as doctors, lawyers, and engineers directly with frontier AI labs for RLHF, evaluation, and specialized training data at massive scale. See all Mercor jobs.
Apply for this role
Your details first, then a short screening test. Both are read as part of your application and go to Mercor, and we will tell you when similar roles appear.