Senior Engineering Manager, Agentic & Generative AI Benchmarking and Evaluations

About the Position

Join ServiceNow as a Senior Engineering Manager for AI Benchmarking and Evaluations, leading AI evaluation practices for enterprise workflows.

Responsibilities

• Build the Evaluation Infrastructure: Design, own, and scale automated testing and evaluation harnesses (unit evals, integration evals, and production drift monitors) to measure agent quality and eliminate regressions.
• Define Enterprise AI Benchmarks: Create standard, repeatable evaluation frameworks tailored to complex business workflows-assessing multi-agent orchestration, intent routing, multi-step planning loops, and long-term memory accuracy.
• Validate Grounding & RAG Pipelines: Partner with search and data fabric teams to systematically evaluate Retrieval-Augmented Generation (RAG) pipelines, hybrid search, and semantic re-ranking systems.
• Model Selection Optimization: Rigorously benchmark frontier LLMs (e.g., OpenAI, Anthropic, Google, and proprietary ServiceNow models) to evaluate trade-offs across execution capabilities, latency, context-window efficiency, and inference costs.
• Lead a High-Performing Team: Recruit, mentor, and foster an AI-native engineering team, driving engineering best practices, prompt-infrastructure stability, and production-grade rigor.
• Cross-Functional Leadership: Collaborate with Core Product, Machine Learning Platforms, and Engineering leads to translate baseline performance statistics into actionable product improvements and model fine-tuning targets.

Requirements

• 8+ years of professional software engineering or machine learning experience, including 3+ years managing or technically leading high-performing AI/ML teams.
• Experience in leveraging or critically thinking about how to integrate AI into work processes, decision-making, or problem-solving.
• Strong foundational knowledge of frontier AI SDKs and deep experience deploying or testing agentic/probabilistic software architectures (multi-agent orchestration, tool execution, and probabilistic feature deployment).
• Demonstrated experience implementing rigorous AI metrics (e.g., ROUGE, BLEU, G-Eval, LLM-as-a-judge patterns, and custom deterministic evaluation code) at an enterprise scale.
• Proficiency in Python and familiarity with data analytics infrastructures (SQL, Pandas, NumPy) alongside standard MLOps tracking platforms.
• Experience with complex knowledge infrastructure, SaaS platform architectures, or relational datasets (e.g., Knowledge Graphs, CMDBs).
• Ability to translate deeply technical evaluation data into executive-level risk assessments, ROI summaries, and strategic roadmap recommendations.
• Bachelor’s or higher degree in Computer Science, Data Science, Machine Learning, or a highly quantitative field (Master's or Ph.D. is a plus).

We Offer

• Base pay of $201,300 - $352,300, plus equity (when applicable), variable/incentive compensation and benefits.
• Health plans, including flexible spending accounts, a 401(k) Plan with company match, ESPP, matching donations, a flexible time away plan, and family leave programs.

About the Company

ServiceNow is the AI control tower for business reinvention. Our ServiceNow AI platform brings together any AI, any data, and any workflow-helping 85% of the Fortune 500® work smarter, faster, and better. We're building an AI-native culture where technology and talent are unstoppable together.

Equal Opportunity Employer

ServiceNow is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, national origin, age, disability, gender identity, veteran status, or any other category protected by law.

Accommodations

We strive to create an accessible and inclusive experience for all candidates. If you require a reasonable accommodation to complete any part of the application process, or are unable to use this online application and need an alternative method to apply, please contact [email protected] for assistance.

Export Control Regulations

For positions requiring access to controlled technology subject to export control regulations, including the U.S. Export Administration Regulations (EAR), ServiceNow may be required to obtain export control approval from government authorities for certain individuals. All employment is contingent upon ServiceNow obtaining any export license or other approval that may be required by relevant export control authorities.