Senior QA & Evaluation Engineer

About the role

Coforge is seeking a Senior QA & Evaluation Engineer to lead the design and automation of testing solutions for both traditional software and emerging AI systems. The role combines deep expertise in test automation with the ability to evaluate large language model performance, ensuring high‑quality releases across multiple Latin American locations.

Key responsibilities

• Design, develop, and maintain automated test suites covering unit, integration, API, and end‑to‑end scenarios.
• Integrate automated tests into CI/CD pipelines and use them as release quality gates.
• Build evaluation frameworks for AI capabilities, including scored rubrics, LLM‑as‑a‑judge approaches, and normalized pass thresholds.
• Maintain held‑out evaluation datasets, measure AI behavior against ground truth, and report defect escape and evaluation metrics.

Required profile

• 4+ years of experience in Quality Engineering with a strong focus on test automation.
• Hands‑on experience with automated testing frameworks, preferably within the .NET ecosystem.
• Strong background in API testing, CI/CD integration, and automated test development.
• Understanding of testing strategies for non‑deterministic and probabilistic AI systems.
• Bachelor’s degree in Computer Science, Software Engineering, Information Technology, or equivalent professional experience.

Required skills

• Test Automation.NET, API Testing, CI/CD, Automated Testing, Test Frameworks.
• LLM Evaluation, AI Testing, evaluation dataset creation, scored rubrics.
• Tools such as Azure AI Foundry Evaluators, promptfoo, Ragas, DeepEval.