Remote opportunity at
SallaSenior AI Backend Engineer - Agent Evaluation & Quality
This remote job at Salla centers on the evaluation tooling for production multi-agent AI systems that serve a large user base. The overriding question is trust:…
Career Tools
About This Role
This remote job at Salla centers on the evaluation tooling for production multi-agent AI systems that serve a large user base. The overriding question is trust: how well each agent performs, whether a change has broken something before release, and how to keep quality stable as versions ship. The engineer builds the evaluation stack: LLM-as-judge systems checked against human labels, evaluation harnesses that run on each pull request, regression detection…
Job Description
This remote job at Salla centers on the evaluation tooling for production multi-agent AI systems that serve a large user base. The overriding question is trust: how well each agent performs, whether a change has broken something before release, and how to keep quality stable as versions ship.
The engineer builds the evaluation stack: LLM-as-judge systems checked against human labels, evaluation harnesses that run on each pull request, regression detection integrated with CI, and user simulators that create test scenarios and adversarial inputs. Real production failures are fed back into evaluation datasets, and the engineer works with product teams to convert quality goals into measurable criteria.
The posting also lets the engineer contribute to agent development itself, beginning with the components they know best from evaluation work. Applicants should have five or more years in software engineering, recent hands-on work with LLM agents, and production experience handling reliability, latency, spend and observability.
Responsibilities
- Own the evaluation stack, including LLM-as-judge systems calibrated against human labels
- Build evaluation harnesses that run on every pull request, with regression detection in CI
- Create user simulators that generate test scenarios and stress cases before real users see them
- Route real production failures into evaluation sets
- Translate product expectations about quality into measurable criteria
- Help improve the agents, beginning with the components that evaluation exposes
Requirements
- Solid software engineering basics, with production Python or TypeScript
- Clean API and system design, plus testing and CI/CD skills
- Practical LLM and agent work, such as retrieval-augmented generation, tool calling and orchestration frameworks like LangGraph or LangChain
- A measurement-first outlook, comfortable with metrics, calibration and experiments
- Operating LLM systems in production with attention to reliability, latency, spend and observability
- Five-plus years in software engineering, recently with LLM or agent projects
Qualifications
- Prior work on offline and online evaluation, LLM-as-judge methods or systematic regression testing
- Use of observability tools such as Arize or LangSmith
- Arabic language or NLP background
- Exposure to merchant-facing products in online retail
Core Skills
Frequently Asked Questions
Answers are based only on the employer’s listing; where it doesn’t say, neither do we.
Is this role remote?
Yes. The listing is remote and the role is full-time. The posting does not name a specific city or country.
What is the focus of the role?
The focus is building evaluation systems for production agents, including LLM-as-judge systems, regression detection in CI and user simulators. The posting also allows contributions to agent development.
Does the posting state a salary?
No. The posting does not specify a salary or any benefits.
Sample Interview Questions
AI-generated questions tailored to this specific role — a preview of the full practice set.
