Researcher, Benchmark Reviews

Key Responsibilities

Review benchmarks: Assess a new benchmark at least every two weeks. You’ll evaluate their methodology and what a “good performance” would imply about AI capabilities.
Publish and maintain reports: Write up public-facing reports on new benchmarks and periodically update them as new versions release and models make progress.
Dig into data: Examine individual tasks within benchmarks in detail, using coding agents to assist while retaining your own critical oversight.

What we're looking for

You excel at written communication. Your first drafts rarely need much polishing; they’re clear and publication-quality with minimal revision.
You think critically about research methodologies. You have strong opinions about what benchmarks and evaluations should look like, and can recognize when something seems wrong.
You’re already familiar with existing AI benchmarks, their methodologies, strengths, and weaknesses. You have ideas for new benchmarks that don’t exist yet, but should.
You’re comfortable working with dense data. You’ll often be breaking down benchmarks into their component parts, which can be overwhelming without familiarity.

Nice to have

Experience writing about AI/ML for a public or research audience.
Hands-on experience running or building AI evaluations.
If you don't tick all these boxes but think you would be a great fit, please consider applying anyway!

Compensation & Benefits

Annual salary between $100,000 and $200,000 USD, depending on location, seniority, and experience.
Salaries are not restricted to USD, and contracts and payments are usually in local currencies. Conversions are based on one-year average exchange rates.
Fully remote environment, including flexible work hours.
Competitive global benefits program, including a comprehensive health insurance program—including supplemental benefits specific to a local country, as available and mandated by local law—and life insurance and a pension plan, if applicable in your country.
Generous paid time off (PTO), including no specific annual limit, with 30 days PTO per year protected, unlimited personal and sick leave, and 4 months paid parental leave for permanent staff with at least 12 months of tenure (prorated parental leave if less than 12 months).
A flexible and generous expense policy for you to spend on equipment and a large range of productivity tools or learning/development opportunities, including unlimited spending on AI tools, subject to regulations and manager approval.
Paid work trips, including 3 staff retreats per year and relevant conferences.
Access to our very well-equipped offices in Berkeley, California, including paid meals, snacks, gym, and more. All staff, independently of where they are based, have access to the office for at least 20 days each year.

Originally posted on Himalayas