AI Developer (LLM Evaluation)
Primary stack
Nice to have's
Job description
About the Position
IT Craft is looking for an experienced AI Developer with hands-on expertise in LLM evaluation and automated testing to join our remote team. In this role, you will take ownership of designing and implementing an evaluation framework for a deployed AI assistant running on AWS Bedrock, with infrastructure support from the internal DevOps team.
Responsibilities
- Design and implement automated end-to-end and regression testing infrastructure for an LLM-based assistant;
- Build an evaluation harness that runs a structured set of prompt scenarios against the deployed service;
- Implement automated checks for response correctness, tool usage, and guardrail behavior;
- Generate clear pass/fail results based on agreed evaluation baselines;
- Collaborate with the internal DevOps team to integrate evaluation runs into CI/CD processes and identify quality and behavior regressions before and after deployments;
- Analyze evaluation results and continuously improve test coverage and reliability;
- Plan and organize the implementation independently, define follow-up tasks, and coordinate with internal specialists through regular synchronization meetings.
Requirements
- Hands-on experience with LLM evaluation tools such as Promptfoo, Langfuse, Arize, or comparable solutions;
- Practical experience designing or implementing automated evaluation and regression testing for LLM-based applications;
- Understanding of approaches used to evaluate non-deterministic LLM responses, tool calls, and guardrail behavior;
- Solid experience with AWS and CI/CD;
- Ability to design a solution from scratch and deliver it independently;
- Familiarity with Kotlin and Spring would be a plus;
- Strong written and spoken English (Upper-Intermediate+ / Advanced).
We Offer
- Remote work opportunity
About the Company
IT Craft is a company that values innovation and collaboration. We are committed to providing our team with a flexible and supportive work environment.
Contact
Recruiter
Nataliia Fesenko
You May Find Interesting
© IT Craft. This job description was sourced from the employer's public career page. TheJob is not the employer — we index the posting and route candidates to the source. All content rights and hiring decisions belong to the employer.
IT outsourcing company