AI Analyst

About the Position

We are seeking a detail-obsessed, data-driven AI Analyst to ensure the accuracy and reliability of our Sales & Finance AI solutions. You will serve as the guardian of accuracy for our Sales & Finance AI chat assistants, ensuring every metric the AI surfaces, especially Bookings and Revenue, is accurate, consistent, and trustworthy.

Responsibilities

Requirements & Solution Definition

• Gather and document business requirements from Finance users and stakeholders through interviews, workshops, observation, and feedback analysis.
• Write clear, detailed user stories with well-defined acceptance criteria that IT can act on without ambiguity.
• Ensure requirements reflect metric integrity standards and executive-readiness expectations.

AI Quality Assurance

• Develop and maintain AI-specific testing frameworks - define test scenarios, expected outputs, edge cases, and pass/fail criteria tailored to conversational AI behavior.
• Establish and document quality standards for AI-generated responses - accuracy, completeness, formatting, tone, and contextual relevance.
• Validate AI-generated responses against Snowflake source data using SQL queries to confirm metric accuracy and consistency.
• Conduct thorough UAT, regression testing, and release testing for every new feature, enhancement, prompt update, data refresh, or model change before go-live.
• Confirm metric integrity - verify that Bookings and Revenue figures surfaced by the AI match official Finance reporting.
• Coordinate business sign-off activities - manage the approval workflow with Finance stakeholders before releases are promoted to production.

Hallucination Prevention

• Design and execute validation test strategies specifically targeting hallucination risks - test for fabricated data, incorrect metric associations, misleading trends, and unsupported narratives.
• Identify and catalog high-risk Finance use cases where hallucination potential is elevated (e.g., multi-step calculations, year-over-year comparisons, segment-level breakdowns, forecast vs. actuals).
• Monitor response quality on an ongoing basis - conduct periodic spot-checks and audits of AI outputs across different query types and user scenarios.
• Escalate and resolve quality concerns immediately - flag suspected hallucinations to the Data Accuracy Specialist (IC4) and team lead; coordinate with IT/Engineering to implement fixes and guardrails.

Executive Use Cases

• Support in designing and testing executive-focused use cases - ensure AI responses meet the precision, tone, and contextual depth expected by C-suite and SVP users.
• Test and validate executive prompt frameworks - verify that curated queries, guided flows, and personalized dashboards produce accurate, polished outputs.
• Contribute to improving usability and decision-support capabilities - identify opportunities to make the executive experience more intuitive, actionable, and insightful.

Adoption & Change Management

• Develop and execute adoption strategies - create structured plans to drive Finance user engagement, from initial awareness through active daily use.
• Lead user onboarding - design and deliver onboarding experiences for new users, including guided walkthroughs, sandbox environments, and office hours.
• Deliver training programs - conduct live demos, workshops, and drop-in sessions tailored to different user groups (analysts, managers, directors).
• Manage communications - draft and distribute release notes, feature announcements, tips & tricks, known issue updates, and adoption progress reports to Finance stakeholders.
• Collect and synthesize user feedback through surveys, interviews, and usage analytics to continuously improve the experience.

Requirements

• Bachelor's degree Required
• 7+ years in Business Analysis / Finance Analytics / FP&A Operations
• Experience with AI-powered tools, chatbots, or conversational AI interfaces
• Understanding of Finance metrics and reporting (Bookings, Revenue, or related areas)
• Working knowledge of SQL and Snowflake
• Understanding of AI/LLM fundamentals - how large language models retrieve, process, and present information; awareness of common failure modes (hallucination, data staleness, context window limitations)
• Prompt Engineering Certificate or related experience

Nice to Have

• Familiarity with Abacus or similar platforms
• Hands-on experience with prompt engineering - designing, testing, and refining prompts to improve AI response quality and accuracy
• Experience creating training materials, user guides, or knowledge base content
• Experience with AI testing frameworks or quality assurance for data-driven applications
• Familiarity with AI governance or responsible AI principles

We Offer

• Competitive salary
• Opportunity to work in a dynamic and innovative environment
• Professional growth and development opportunities

About the Company

[Company description if present]