Remote opportunity at
AnaplanQA Automation Engineer - AI
Anaplan is seeking a QA Automation Engineer to focus on quality assurance for Generative AI and agentic systems. The company provides a scenario planning and analysis…
Career Tools
About This Role
Anaplan is seeking a QA Automation Engineer to focus on quality assurance for Generative AI and agentic systems. The company provides a scenario planning and analysis platform utilized by organizations such as Coca-Cola, LinkedIn, Adobe, LVMH, and Bayer. In this newly created position, the professional will build evaluation frameworks, testing strategies, and quality metrics specifically for applications powered by large language models. The role requires ensuring that AI features remain…
Job Description
Anaplan is seeking a QA Automation Engineer to focus on quality assurance for Generative AI and agentic systems. The company provides a scenario planning and analysis platform utilized by organizations such as Coca-Cola, LinkedIn, Adobe, LVMH, and Bayer. In this newly created position, the professional will build evaluation frameworks, testing strategies, and quality metrics specifically for applications powered by large language models. The role requires ensuring that AI features remain accurate, reliable, and trustworthy through comprehensive automated testing.
This position is suited for an engineer who combines traditional testing expertise with a strong understanding of artificial intelligence behavior. Responsibilities include running end-to-end automated tests, performing regression testing for prompts, supporting evaluation frameworks, curating test datasets, and assisting with adversarial testing to detect biases or model hallucinations. The ideal candidate will also help developers validate their code and take ownership of automated coverage within their designated workstream.
Anaplan operates with a commitment to diversity, equity, inclusion, and belonging, welcoming individuals from all backgrounds. The organization provides reasonable accommodations for candidates and employees with disabilities. Applicants are encouraged to remain vigilant against recruitment fraud and verify that communications originate from official company channels.
Responsibilities
- Develop and execute end-to-end automated test coverage across evaluations, APIs, UIs, and performance tests using Python
- Implement testing strategies for GenAI features including conversational AI, agentic systems, and LLM-powered workflows
- Run and maintain automated test suites for prompt testing and regression checks to detect model behavior changes
- Maintain evaluation frameworks that measure GenAI performance regarding accuracy, relevance, safety, consistency, and latency
- Curate high-quality test datasets and golden examples representing diverse scenarios and edge cases
- Conduct adversarial testing to spot potential model hallucinations or biases and support production alerting systems
- Utilize internal tools to help developers validate GenAI code and assist with user acceptance testing
- Document testing procedures and known issues clearly for the assigned workstream
Requirements
- Test engineering or QA experience with exposure to or an interest in AI and machine learning systems
- Hands-on coding experience using Python
- Understanding of GenAI technologies such as LLMs, prompt engineering, and AI application patterns
- Familiarity with test automation frameworks and scripting including Python, Pytest, API, and UI testing
- Knowledge of software testing methodologies covering functional, integration, regression, performance, and security testing
- Familiarity with API testing tools like Postman and REST Assured alongside backend testing
- Knowledge of CI/CD pipelines and automated testing integration
Core Skills
Frequently Asked Questions
Answers are based only on the employer’s listing; where it doesn’t say, neither do we.
Is this position remote?
Yes, the source posting lists the remote status as remote.
What is the employment type for this role?
The posting specifies the employment type as full-time.
What salary does Anaplan offer for this position?
The salary is not stated in the source posting.
What core technical skills are required?
Candidates must have hands-on Python coding experience, QA or test engineering experience with AI systems, knowledge of LLMs and prompt engineering, and familiarity with test automation frameworks and software testing methodologies.
Sample Interview Questions
AI-generated questions tailored to this specific role — a preview of the full practice set.