Data Scientist

About the Position

Data Scientist

Location: Amsterdam, Noord-Holland, Netherlands

Employment Type: FULL_TIME

Work Location: On-site

---

Responsibilities

At Kaizo, we are seeking a sharp, analytical Data Scientist to join our team and own the evaluation and quality loop of our AutoQA product. Your role will involve:

• Translate customer rubrics into AutoQA instructions: Work with real customer quality criteria and turn them into precise, testable instructions that LLMs can score reliably.
• Run experiments that move accuracy: Design and execute evaluation experiments on large, representative datasets using LangSmith and BigQuery, and track quality with our performance metrics.
• Build the datasets that make evaluation possible: Curate raw production data into golden datasets with balanced coverage, and generate synthetic data to cover the rare cases that matter most.
• Build LLM-as-a-judge pipelines: To assess system quality internally and make evaluation repeatable.
• Make results actionable: Your experiments should end in a decision: change a prompt, adjust which tools the system uses, surface context the AI is missing, or flag where new capabilities are needed. You'll work with the AI team to ship those decisions.
• Evaluate across the full stack: Beyond scoring quality, you'll help validate retrieval (RAG/IR), tool calling, and speech pipelines (transcription and diarization quality).
• Join customer calls with the team: To understand how QA leaders define quality, and feed what you learn back into the product.

---

Requirements

• A solid understanding of how LLMs work and hands-on experience prompting them for accuracy (coursework, thesis, internships, or side projects all count; production experience is a bonus).
• Good applied statistics: experiment design, classifier evaluation, precision/recall trade-offs, and working with heavily imbalanced data.
• Strong Python skills and fluency with the standard data toolkit (Pandas, NumPy, Jupyter). SQL is a plus.
• An analytical, evidence-first mindset: you'd rather measure than assume.
• Product sense and empathy for end users. You'll be building for QA managers and support agents, not just for benchmarks.
• Excellent written and verbal communication. You'll present findings to the team and join customer conversations.
• 0 to 2 years of industry experience. Recent graduates with strong relevant work are encouraged to apply.
• A team player who's comfortable wearing multiple hats. We're an early-stage company and things move fast.

---

Nice to Have

• Experience with LangSmith or similar LLMOps/evaluation tooling
• Google Cloud Platform, BigQuery, Docker, or Kubernetes
• Speech/audio processing or ASR evaluation
• Fine-tuning or building synthetic datasets for LLMs

---

We Offer

• An office in the heart of Amsterdam, with the flexibility of hybrid working
• Visa sponsorship available for eligible candidates
• Great office gear: MacBook, tools, desk, chair, whatever you need
• Flexible working schedule and an open holiday policy
• Fun workations and team events
• A clear growth path into Senior Data Scientist, AI Engineer, or ML Engineer roles

---

About the Company

At Kaizo, we build a performance development and quality platform for customer support teams. Our AutoQA product uses LLMs to review support conversations against each customer's own quality rubric, automatically and at scale. Behind it sits a microservices-based stream processing platform handling over 200 million events per day (Kafka, Kubernetes on Google Cloud, ElasticSearch, MongoDB, BigQuery), and an LLMOps stack built around LangSmith for experimentation, prompt management, and tracing.

The hard part isn't calling an LLM. It's knowing, with evidence, how well the system performs on every customer's unique rubric, and having a reliable, repeatable way to improve it. That's where you come in.