AI Software Test Engineer

AI Software Test Engineer role in Ann Arbor, MI (onsite), focused on validating AI-powered applications for reliability, accuracy, safety, and performance.

Responsibilities

• Design and implement testing strategies across a portfolio of AI applications, platforms, and services.
• Assess AI model outputs for accuracy, consistency, reliability, and safety to support high-quality user experiences.
• Plan and execute functional, integration, end-to-end, regression, and performance testing for AI solutions.
• Create test cases tailored to prompt-driven, agentic, and retrieval-based AI workflows.
• Validate AI guardrails, business rules, permissions, and governance controls for responsible AI deployment.
• Run adversarial, negative, and edge-case tests to surface potential model failures and hallucinations.
• Build and maintain automated test frameworks for AI applications to streamline testing.
• Develop automated evaluation pipelines to continuously assess AI responses and workflows.
• Integrate AI testing into CI/CD pipelines to enable continuous quality and fast iteration.
• Implement automated quality scoring and regression detection to maintain release standards.
• Create reusable test data, mocks, simulators, and validation frameworks to improve efficiency and coverage.
• Test AI agents, workflows, APIs, and tool-calling capabilities to verify correct operation.
• Validate integrations with external systems, diverse data sources, and enterprise services.
• Verify performance, reliability, scalability, and resiliency of AI workloads through rigorous testing.
• Execute load and stress testing to confirm AI services handle anticipated demand.
• Collaborate with software engineers, AI engineers, product owners, architects, and security teams across the development lifecycle.
• Participate in design reviews and provide quality feedback early in development.
• Help define test strategy, quality standards, and best practices for the organization.
• Support production readiness reviews and lead defect triage to help ensure timely issue resolution.

Requirements

• Bachelor’s degree in Computer Science, Software Engineering, Information Systems, or a related technical field.
• 3 to 6 years of progressive experience in software testing, QA automation, or quality engineering.
• Experience building automated test solutions using one or more: Python, Java, JavaScript/TypeScript, C#.
• Demonstrated experience with API testing and automation tools (for example, Postman).
• Strong foundation in test automation principles and best practices.
• Working knowledge of SDLC and Agile methodologies.
• Experience integrating testing into CI/CD pipelines (for example, GitHub Actions, Jenkins).
• Experience testing distributed systems, web applications, and APIs.
• Familiarity with containerization and orchestration, including Docker and Kubernetes.
• Experience testing on cloud platforms: Azure, AWS, or GCP.
• Proficiency in quality engineering concepts such as Defect Analysis and Root Cause Investigation.

Technologies

• Python
• Java
• JavaScript/TypeScript
• C#
• Postman
• GitHub Actions
• Jenkins
• Docker
• Kubernetes
• Azure
• AWS
• GCP
• Selenium
• Playwright
• JUnit
• PyTest

Benefits

• Choice of three major medical plans (including vision coverage) plus dental insurance
• Paid sick leave for all hourly talent in the U.S. and Canada
• 401(k) with a match and immediate vesting
• Exclusive employee discounts through Promo Code: Aquent
• Free online training through Aquent Gymnasium

Preferred Qualifications (Nice-to-Have)

• Experience testing generative AI applications, LLM-based systems, AI agents, or RAG applications.
• Familiarity with AI models and platforms such as OpenAI, Claude, Gemini, or Azure OpenAI.
• Experience building evaluation and benchmarking frameworks for AI solutions.
• Knowledge of responsible AI principles, AI governance frameworks, and AI risk management practices.
• Experience with AI-specific testing techniques including Prompt Testing, Response Evaluation, Hallucination Detection, Agent Workflow Validation, RAG Validation, AI Safety Testing, AI Regression Testing, and AI Benchmarking.
• Familiarity with automation frameworks like Selenium, Playwright, JUnit, or PyTest.
• Exposure to quality engineering domains such as Performance Testing, Load Testing, and Security Testing.

Location: Ann Arbor, MI (onsite)

Compensation: USD 42 - 46 per hourly