About the role
ABOUT THE ROLE
Lead the research engineering work focused on measuring and improving training data for AI agents. You will shape the strategy and systems used to evaluate data across reinforcement learning environments, synthetic datasets, benchmarks, and domain-specific workflows.
WHAT YOU'LL DO
- Lead a team building systems to evaluate large volumes of tasks across environments, datasets, benchmarks, and specialized workflows.
- Define data quality strategy by creating quality-control systems, setting standards, and designing experiments to assess agent outputs.
- Develop scalable approaches to validate synthetic data, including failure-mode analysis, task mutation checks, and trajectory auditing.
- Work with research engineers, domain experts, and data vendors to diagnose quality problems and improve data generation.
- Turn research findings into production tools, dashboards, validation pipelines, and feedback loops.
- Help establish rigorous standards for assessing whether agent training data is useful, and mentor research engineers.
WHAT WE'RE LOOKING FOR
- At least 5 years of research or engineering experience building AI or machine learning data evaluation and quality systems.
- Experience leading technical teams or projects from problem definition through implementation and iteration.
- Advanced proficiency in Python, Docker, and Linux environments.
- Strong background in AI evaluations, post-training, synthetic data validation, benchmarks, or related evaluation infrastructure.
- Sound judgment about what makes training tasks realistic, learnable, diverse, reliable, and useful.
- Experience translating research insights into production systems and pipelines, and designing metrics, experiments, and quality processes.
- Ability to work with subject-matter experts to turn domain judgment into scalable review or data generation systems.
- Clear written communication, careful attention to detail, and experience in an early-stage startup environment.
COMPENSATION & BENEFITS
Base salary: $130,000 to $225,000 annually. Visa sponsorship is available.
LOCATION
On-site in the United States.