AI Benchmark Quality Reviewer (Remote Contractor)

About the role

Help review the quality and fairness of challenging tasks used to evaluate AI systems. You will inspect task instructions, model execution traces and grading behavior, then explain whether a result reflects genuine model performance or an issue with the task, grader or environment.

Key responsibilities

• Check task instructions, source materials, reference solutions and evaluation criteria for consistency and completeness.
• Review model execution traces, tool calls and deliverables to assess whether successes and failures are justified.
• Identify brittle grading checks, unsupported criteria and valid alternative solutions that may have been marked incorrect.
• Investigate discrepancies and distinguish model limitations from task, grader, tool or environment issues.
• Write concise, evidence‑backed findings and verify that revisions address the issues found.

Required profile

• At least five years of relevant technical or analytical experience.
• Strong written English, analytical judgment and attention to detail.
• Ability to give specific, reproducible feedback and explain uncertainty clearly.
• Experience in AI evaluation, technical QA, data analysis or benchmark development is helpful but not required.
• Familiarity with Harbor task setup is a plus.

Required skills

• Python
• SQL
• Shell scripts
• Structured data analysis
• Execution log interpretation

What we offer

• Compensation at $23 per hour.
• Eight‑week remote contract, 40 hours per week with eight hours of daily overlap with Pacific Time.