Remote opportunity at
TuringSenior Software Engineer, AI Evaluation Infrastructure
Turing is seeking a Senior Software Engineer for its AI Evaluation Infrastructure team. Headquartered in San Francisco, Turing collaborates with frontier artificial intelligence laboratories and Fortune…
Career Tools
About This Role
Turing is seeking a Senior Software Engineer for its AI Evaluation Infrastructure team. Headquartered in San Francisco, Turing collaborates with frontier artificial intelligence laboratories and Fortune 500 companies to generate specialized datasets, reinforcement learning environments, and benchmarks for software engineering, enterprise tasks, and advanced STEM reasoning. In this position, you will build and maintain the infrastructure used by frontier AI labs to assess coding agents and language models at scale.…
Job Description
Turing is seeking a Senior Software Engineer for its AI Evaluation Infrastructure team. Headquartered in San Francisco, Turing collaborates with frontier artificial intelligence laboratories and Fortune 500 companies to generate specialized datasets, reinforcement learning environments, and benchmarks for software engineering, enterprise tasks, and advanced STEM reasoning. In this position, you will build and maintain the infrastructure used by frontier AI labs to assess coding agents and language models at scale. You will own features across the entire technology stack, handling data models, application programming interfaces, background jobs, and user interfaces. This role is intended for an autonomous engineer who excels at turning high-level requests into finished, tested products with sound technical tradeoffs.
Your daily work will involve executing trial runs in sandboxed environments, aggregating thousands of executions into reliable metrics, and refining automated quality checks through verifiers and rubric or judge-based grading. You will integrate new language models, agents, and benchmarks generally without custom exceptions while keeping queries efficient and systems reliable under high concurrency. Operating with startup speed, you will adhere to written engineering guidelines, favor simple architectures, and maintain clean boundaries between services.
Responsibilities
- Design and ship end-to-end features spanning data models, application programming interfaces, background workers, and user interfaces
- Build and improve evaluation pipelines for language models and agents, including running sandboxed trials and scoring results
- Enhance benchmark task quality checks using verifiers, rubric-based grading, and model-based judges
- Integrate new models, agents, and benchmarks in a generalizable manner without individual special cases
- Maintain system speed and reliability as usage scales through efficient database queries, safe migrations, and robust concurrency handling
- Review peer pull requests to ensure alignment with engineering guidelines and codebase consistency
Requirements
- Minimum of five years of experience building and operating production backend systems with ownership of at least one service from design through on-call
- Strong proficiency in Python development
- Familiarity with Django or a similar web framework, PostgreSQL, and task queues such as Celery
- Hands-on experience in language model evaluation beyond simple API calls, including benchmark task construction, trial scoring, and result aggregation
- Strong engineering judgment regarding task quality, defensive verifier design, and making minimal design tradeoffs
- Ability to independently take a vague request, locate relevant code, and complete implementation including tests and migrations
- Comfort working with TypeScript and React frontends when interface changes are necessary
- Clear written communication skills for pull request descriptions and design notes
Qualifications
- Experience running agents in sandboxed or containerized environments using Docker, Kubernetes, or cloud sandboxes like Modal or Daytona
- Experience with model-based or rubric-based grading and minimizing judge bias or noise
- Familiarity with coding agents such as Claude Code, Codex, or OpenHands, as well as the Model Context Protocol
- Experience managing distributed job scheduling, concurrency limits, or large-scale database locking
- Prior contributions to open-source evaluation or infrastructure projects
Core Skills
Frequently Asked Questions
Answers are based only on the employer’s listing; where it doesn’t say, neither do we.
What is the remote work policy for this position?
The posting specifies that the role is remote, but does not state any further location or regional restrictions beyond mentioning Turing's headquarters in San Francisco.
What are the primary technical requirements for the role?
Candidates must have at least five years of experience building and operating production backend systems, strong Python skills, and experience with relational databases, task queues, and language model evaluation.
What is the employment type?
The job posting indicates that this is a full-time position.
What is the salary range offered?
The salary for this position is not stated in the job posting.
Sample Interview Questions
AI-generated questions tailored to this specific role — a preview of the full practice set.