Job Description, Responsibilities & Requirements
About the Position
We are seeking an AI Quality Assurance Engineer for AI product development to join our team in Tokyo, focusing on ensuring the quality and reliability of AI products.
About the Company
Woven by Toyota is contributing to Toyota's transformation into a mobility company once in a century. Inspired by Toyota's history of inventing "for someone else," our mission is to expand the definition of mobility and the ways it can contribute to society, challenging the conventional wisdom of mobility with people-focused innovation.
We operate around four pillars: AD/ADAS technology for autonomous and advanced driver-assistance systems, Arene, a vehicle software production platform for SDV (Software-Defined Vehicle), Woven City, a mobility test course, and Cloud & AI, the digital infrastructure supporting the collaborative platform. Together, we strive to achieve a "zero-accident society" and the "mass production of happiness."
Team Overview
Toyota is redefining the meaning of mobility. We move people, things, information, and energy, continually challenging the conventional wisdom of mobility. Woven City, as a test course for creating and testing various types of mobility, aims for the mass production of happiness with the concept of a people-centered city, a city of real experiments, and an unfinished city.
Woven City brings together diverse colleagues who share a vision for future mobility. Inventors, residents, and visitors will live and demonstrate various products and services in a test course shaped like a city where people, mobility, and social infrastructure are interconnected.
For more information about Woven City, please visit: Woven City
Job Description
As an AI Product Quality Assurance Engineer, you will be responsible for ensuring the quality of AI products, focusing on aspects such as accuracy, consistency, safety, fairness, and UX. Your role will involve more than just conducting one-time tests; you will also be tasked with standardizing and automating evaluation flows to continuously improve quality based on user feedback and evaluation results.
You will work closely with product development teams, MLOPS, and data engineers to redesign, implement, and operate feedback loops from a QA perspective. Specifically, you are expected to lead the initial feedback loop driven by QA, even in situations where real user feedback is insufficient.
Your duties will include:
- Ensuring the quality of AI products (accuracy, consistency, safety, fairness, UX, etc.)
- Conducting scenario testing, exploratory testing, and red teaming
- Analyzing trends in LLM outputs and identifying failure patterns
- Designing quality evaluations using user feedback (logs, ratings, inquiries, etc.)
- Standardizing and organizing evaluation processes in a reusable format
- Automating quality evaluation flows where possible
- Collecting and aggregating evaluation data
- Establishing and operating feedback loops in collaboration with development, MLOPS, and data engineers
- Documenting and sharing quality issues and risks with the team
Requirements
- 3+ years of experience in QA/testing of products using LLM
- Experience in test design (organizing perspectives, creating test cases)
- Ability to establish evaluation criteria for ambiguous specifications or targets with no unique correct answer
- Experience in automating tests and evaluation processes
- Experience in improving quality in collaboration with development teams
- Business-level proficiency in both English and Japanese
Nice to Have
- Practical experience in scenario testing and red teaming
- Experience in evaluating and analyzing user feedback and logs
- Proactive attitude in identifying and addressing challenges
- Basic understanding of natural language processing, particularly in Japanese
We Offer
- Competitive salary based on industry standards and experience
- Flexible working hours
- Annual paid leave: 20 days (varies depending on the month of joining in the first year)
- Sick leave: 6 days annually (varies depending on the month of joining in the first year)
- Holidays: Saturdays, Sundays, public holidays, and other days as designated by the company
- Social insurance: Health insurance, pension insurance, work injury insurance, employment insurance, and nursing care insurance
- Housing allowance
- Retirement benefits
- Car rental support
- In-house training programs (software and language learning)
Our Commitment
- We are committed to equal employment opportunities and respect diversity.
- Personal information provided will be used solely for recruitment and employment procedures. For more details, please refer to our privacy policy.
Apply for this job
Jobs powered by