Staff Technical Product Manager, Evals
Hercules is looking for a senior or staff level product manager to own evals for our AI coding agent
- Your #1 and only goal is to make Hercules ' AI coding agent cheaper, faster, and more accurate (our pareto frontier)
- You will own Hercules Bench, our offline eval set
- You will own Hercules online evals
- You will leverage our 500k+ users, 1M+ code repos, and millions of chat threads to guide your eval creation
- You will anonymize data to protect our customer's privacy
We expect candidates to be incredibly strong across all of the following areas:
Software fundamentals : You are building an AI coding agent. We expect you to be technical and able to code. We expect you to have a decent understanding of system design, architecture, data modeling, and API design. In saying this, you will work with an engineering counterpart to work on complex implementation
Evals . You have explicitly created and/or managed technical evals before. You should be able to work across a variety of different eval methodologies and deeply understand optimal reward modeling,
Product sense : strong intuition for the goals customers are trying to achieve and novel solutions to help them achieve them
Design sense: Track record of building simple, delightful UX/UI that makes it easy for customers to achieve their goals
AI context and harness engineering : strong experience with the most up to date context and harness