Machine Learning Engineer (Evals and Voice Models)

About the Position

Aircall is a unicorn, AI-powered customer communications platform used by 22,000+ companies worldwide to drive revenue, resolve issues faster, and scale customer-facing teams. We’re redefining customer communications by bringing voice, SMS, WhatsApp, and AI together into one seamless workspace.

We are looking for a Machine Learning Engineer (Evals and Voice Models) to join our team in San Francisco. You'll work on voice models, agent capability evaluations, benchmark design, LLM-as-judge systems, failure analysis, and the infrastructure that ties it together by establishing shared metrics, test sets, and tooling to measure accuracy, resolution quality, and safety consistently across products.

Responsibilities

• Design and document comprehensive evaluation frameworks for Aircall’s AI agents across voice, chat, and messaging.
• Train and fine-tune voice models (TTS, ASR, speech-to-speech) using production and synthetic data.
• Assess AI generated solutions across training pipelines, experimentation setups, debugging processes, and optimization strategies.
• Analyze system design decisions and identify strengths, weaknesses, and potential failure points.
• Design annotation guidelines and workflows for human-labeled evaluation data, and calibrate LLM-as-judge systems against human raters.
• Build and maintain live quality monitoring for deployed AI agents.
• Own the metric contract for every published AI metrics.
• Build release gates, the offline regression suite each AI surface must pass before a prompt, model, or config change ships.
• Build voice-specific evaluation: simulated callers across accents, languages, background noise, barge-in, DTMF, and tool failures.

Requirements

• BS in Computer Science, Machine Learning, Statistics, or related field
• 3+ years of experience in ML Engineering or Applied ML with 8+ years of overall experience
• Strong experience in evaluating supervised, unsupervised, LLMs, and deep learning models.
• Hands-on experience in failure analysis and evaluating LLMs
• Experience building automated evaluation systems
• Strong communication skills to articulate complex technical concepts across technical and non-technical audiences
• Hands-on experience training or fine-tuning voice/speech models (TTS, ASR, or speech-to-speech), including data pipeline construction and experimentation.

Nice to Have

• MS / PhD in Computer Science, Machine Learning, Statistics, or related field
• Experience evaluating LLMs or agentic systems (e.g., LLM-as-a-judge, RAG evaluation)
• Experience with synthetic data generation and prompt engineering
• Experience training or fine-tuning voice models at scale, with familiarity in synthetic data generation, model distillation, or low-latency inference optimization for production voice agents.

We Offer

• Competitive salary package: $181,000 - $250,000 USD
• Key moment to join Aircall in terms of growth and opportunities
• Our people matter, work-life balance is important at Aircall
• Fast-learning environment, entrepreneurial and strong team spirit
• 45+ Nationalities: cosmopolite & multi-cultural mindset

About the Company

At Aircall, we believe diversity, equity, and inclusion – irrespective of origins, identity, background, and orientations – are core to our journey. We pride ourselves on promoting active inclusion within our business to foster a strong sense of belonging for all. We’re working to create a place filled with diverse people who can enrich and learn from one another. We’re committed to ensuring that everyone not only has a seat at the table but is valued and respected at it by providing equal opportunities to develop and thrive.

We will constantly challenge ourselves to make sure that we live up to our ambitions around diversity, equity, and inclusion, and keep this conversation open. Above all else, we understand and acknowledge that we have work to do and much to learn.