AI QA Engineer

Krisp
Krisp

This job is no longer accepting applications

See open jobs at Krisp.See open jobs similar to "AI QA Engineer" Sierra Ventures.

Software Engineering, Data Science, Quality Assurance

Armenia · Remote

Posted on Feb 26, 2026

AI QA Engineer

Armenia
Hybrid

The AI QA Engineer is responsible for developing and executing AI model evaluation strategies, implementing automated and manual testing for LLM-based applications, detecting biases and hallucinations, and collaborating with engineers to optimize model performance and ensure high-quality AI outputs.

What you'll do
  • Develop and execute AI model evaluation pipelines, ensuring accuracy, consistency, and fairness.
  • Implement automated and manual testing for LLM-based applications.
  • Work closely with AI engineers to debug failures, identify root causes, and optimize model performance.
  • Collaborate with AI Engineers to integrate testing into early-stage development.
  • Build and manage test datasets, ensuring high-quality, diverse, and balanced samples.
  • Develop synthetic data pipelines to enhance model evaluation.
  • Perform AI Pipeline integration tests in appropriate environments before releases
  • Develop and maintain test reports, summarizing findings and recommendations.
What we are looking for
  • Experience with AI/ML testing frameworks and LLM evaluation methodologies.
  • Strong understanding of LLM behaviors, biases, failure modes, and edge cases.
  • Proficiency in Python and familiarity with ML testing frameworks (e.g., PyTest, Unittest, custom ML evaluation tools).
  • Experience with test dataset management and annotation tools.
  • Familiarity with synthetic data generation and adversarial testing techniques.
  • Strong problem-solving and debugging skills to analyze AI failures and inconsistencies.
  • Strong English language proficiency with the ability to evaluate AI-generated text and improve prompts.
How to apply

All interested candidates are encouraged to apply through this form.
We highly appreciate all applications, however, only shortlisted candidates will be contacted for the next stages.

Krisp is an Equal Opportunity Employer:

All applicants are considered regardless of race, color, religion, national origin, age, sex, marital status, ancestry, physical or mental disability, veteran status, gender identity, or sexual orientation. We do not tolerate discrimination or harassment of any kind. All employees and contractors of Krisp treat each other with respect and empathy.

This job is no longer accepting applications

See open jobs at Krisp.See open jobs similar to "AI QA Engineer" Sierra Ventures.
See more open positions at Krisp