Software Engineer, QA & Test Automation

About the Position

Software Engineer, QA & Test Automation

Aviary AI is looking for a Software Engineer focused on QA and test automation (90-day probationary period) to own quality across our voice agent and software deployments. You’ll sit at the edge of our deployment cycle with real authority over what ships - starting hands-on with manual QA, then building the tooling that automates it, and ultimately owning the systems that keep our agents reliable as we scale.

Responsibilities

• Own the quality bar for voice agent and software deployments - you decide what ships, what waits, and what needs more work.
• Design and maintain automated test and evaluation suites across our voice agent stack and standard software pipelines.
• Build tooling that replaces manual QA, focusing on automation, scalability, and reliable release gating.
• Take part in the design and architecture of testing and evaluation infrastructure across the product.
• Help shape team culture, internal tools, and the processes around how we ship.

What You’ll Work With

• Aviary Web QA suite - the in-platform testing tools (React/Next.js, TypeScript) covering chat, voice, and adversarial testing of agent configs, with automated grading against success definitions.
• Blue-Jay - our adversarial testing framework (Python) that generates customer personas and runs text and phone battles against agents to probe edge cases and failure modes.
• Coach - our run-to-completion agent (Python) that reviews campaigns, analyzes flagged calls, proposes config changes, and validates them through Blue-Jay before reporting to human reviewers.

Requirements

• BA/BS in Computer Science, Engineering, or a related field.
• 1–3 years of software development experience (including internships); strong new grads will be considered.
• Proficiency in React/Next.js, with experience building responsive, scalable web applications.
• Strong general programming ability - Python and TypeScript both matter, since our stack spans both.
• Familiarity with AWS services (e.g. EC2, S3, Lambda, ECS) and cloud architecture for deploying scalable solutions.
• Familiarity with relational databases, particularly PostgreSQL and SQL Server, including writing and optimizing queries.
• Excellent problem-solving skills and the ability to work independently in a fast-paced startup environment.

Nice to Have

• Experience with test automation, CI, or evaluation frameworks.
• Work in natural language processing, machine learning, recommendation systems, or large-scale production search/ranking systems.
• Prior experience with LLM or voice agent evaluation and adversarial testing.
• UX/UI experience a plus (i.e. Figma).

We Offer

• Equity
• Hybrid work - 3+ days in our NYC office
• Health, dental, and vision coverage
• Unlimited PTO
• In-office meals stipend
• Learning & development budget

About the Company

AviaryAI

Aviary AI transforms the way financial services companies communicate with their customers by using AI technologies. Aviary, an outbound AI sales team for credit unions, banks and insurance companies, provides interactions that are human-like and customized to fit the operational and regulatory requirements of credit unions, banks and insurance providers. Aviary uses a private large language models and industry data to ensure industry compliance and high-quality interactions.

About the Interview

• Initial Call: 15–30 minute intro call with our co-founder/CEO to get to know each other.
• Hiring Manager Call: 30 minute call with the hiring manager.
• Technical Call & Coding Challenge: Meet with our dev team. You’ll submit a link to a recent project, walk us through the code and your logic, and make a small change live. We’ll schedule a 1-hour session with the engineering team - including our co-founder/CTO, who is the hiring manager - to go through your approach.
• Final Interview: 30 minute call/in-person meeting with our founders (CEO, CTO, COO, and Head of Data).
• Reference / background check.
• Offer!