Senior AI QA Automation Engineer

About the role

Coforge is seeking a Senior AI QA Automation Engineer to design and maintain automated evaluation frameworks for AI agents, RAG solutions, and related workflows. The role is fully remote and supports candidates located in Costa Rica, Peru, Colombia, or Bolivia.

Key responsibilities

• Build automated evaluation frameworks for AI agents, RAG solutions, prompts, tools, workflows, and model outputs, including benchmark datasets, golden responses, regression suites, evaluation metrics, and release thresholds.
• Test AI‑specific behaviours such as hallucinations, grounding, citation accuracy, prompt injection, tool selection, and multi‑agent coordination.
• Develop agent‑powered QA solutions for test‑case generation, API and workflow validation, and intelligent UAT, covering UI, API, integration, performance, and end‑to‑end automation.
• Implement CI/CD quality gates and continuously monitor production AI quality, drift, reliability, and recurring failure patterns.

Required profile

• 5+ years of experience in Software Quality Engineering, Test Automation, or Reliability Engineering.
• Strong Python expertise and experience building reusable automated testing frameworks.
• Bachelor’s degree in Computer Science, Software Engineering, Information Systems, or a related field.

Required skills

• Python
• Playwright or Selenium
• API, UI, integration, regression, end‑to‑end, and CI/CD‑based test automation
• Azure AI Foundry Evaluations, RAG evaluation, and prompt testing
• Defining evaluation metrics, benchmark datasets, regression suites, and quality thresholds
• Implementing CI/CD quality gates and production monitoring