Staff Technical Product Manager – AI/LLM expertise + AI Evaluation Science
Prefer a full account with a password? Sign Up to Apply
Job Description
We are looking for a highly technical, strategic, and outcome-oriented Staff Technical Product Manager to establish and scale AI Evaluation as a core engineering capability across the DT organization. The Eval Technical Product Manager will lead this transformation by making trusted Evals the evidence layer for AI.
This is a dual-role position with responsibility for both:
• Platform Eval Strategy & Adoption
Partner closely with Servicenow AI platform and product teams to understand, implement, validate, and drive adoption of out-of-the-box Evals, while influencing the platform roadmap based on enterprise needs.
• Enterprise Eval Innovation
Identify gaps in platform evaluation capabilities and drive the definition and development of custom Evals, metrics, rubrics, datasets, and evaluation strategies for enterprise-specific AI applications and agentic workflows. The role will establish Eval-Driven Development as a standard engineering practice—moving AI quality from point-in-time testing to continuous measurement throughout the AI lifecycle
• Custom Eval Innovation
• Identify gaps where OOTB Evals don't adequately measure enterprise AI quality.
• Define and drive custom metrics, rubrics, datasets and evaluation strategies.
• Develop reusable Evals for domain-specific behaviors and business outcomes.
• Establish evaluation approaches for AI agents, task completion, tool usage and multi-step workflows.
• Eval-Driven Development
• Embed Evals into the full AI lifecycle: Define → Measure → Build → Evaluate → Improve → Release → Monitor
• Require AI teams to define expected behavior and measurable quality criteria.
• Establish evaluation coverage and quality thresholds.
• Integrate trusted Evals into development, regression and release workflows.
• Establish Eval-based quality gates for appropriate AI applications.
• Enable continuous evaluation in production.
Requirements
Department: Engineering, Infrastructure and Operations
Function: Engineering
Experience Level: Not Applicable