Opportunity at
Culture AmpStaff Applied AI Scientist
Culture Amp is seeking a Staff-level role for an Applied AI Scientist, someone with deep AI engineering experience, to raise the quality of its Coach AI…
Career Tools
About This Role
Culture Amp is seeking a Staff-level role for an Applied AI Scientist, someone with deep AI engineering experience, to raise the quality of its Coach AI product after launch. The central challenge is ongoing evaluation of live behaviour: detecting when performance changes, finding the underlying causes, and pushing measurable improvements across a growing system. The person is also expected to make this approach repeatable so other engineering teams can run…
Job Description
Culture Amp is seeking a Staff-level role for an Applied AI Scientist, someone with deep AI engineering experience, to raise the quality of its Coach AI product after launch. The central challenge is ongoing evaluation of live behaviour: detecting when performance changes, finding the underlying causes, and pushing measurable improvements across a growing system. The person is also expected to make this approach repeatable so other engineering teams can run it themselves.
The scientist runs the full improvement cycle, from prompt design through large-scale assessment to refinement. This includes building language-model tools that analyse shifts, produce deeper diagnostics and propose prompt or system changes. The scientist also shapes what enters a model's working context, covering retrieval, memory across conversations, assembly and compression of context, and budgeting context in long multi-step runs. Each change must be justified by evaluation results rather than intuition.
Evaluation work includes sampling, automated judging by language models, and human labelling of de-identified production traces, using tooling such as Langfuse to watch quality over time. The person contributes to agent design, including planning, tool selection, routing and checking steps, and lets evaluation findings drive structural changes. Model and provider choices are owned, weighing quality, speed and cost. Guardrails for sensitive coaching and people data, covering personal information handling, content safety and resistance to misuse, are part of the remit.
Candidates should have built and operated production agent systems, analysed AI products in live use, run evaluations with judges and labelled datasets, and worked with observability tools. Regular use of agentic coding assistants and clear writing are expected. Eligibility to work in Australia is required, and the position is based in Melbourne or Sydney.
Responsibilities
- Own the cycle of prompt refinement, evaluation at scale and ongoing improvement
- Build language-model tools that spot performance changes and recommend fixes
- Design context handling: retrieval, cross-session memory, assembly and compression
- Run evaluations using automated judges and labelling over de-identified traces
- Guide agent architecture on planning, tool use, routing and verification based on findings
- Choose models and providers by weighing quality, latency and cost
- Build guardrails, personal data protections and content-safety controls
- Provide reusable frameworks and documentation so other teams can evaluate their own work
Requirements
- Track record of production agent systems, including context engineering, retrieval-augmented generation, memory and cost management
- Experience analysing live AI or data products and turning results into improvements
- Hands-on production evaluation with judges, evaluation datasets and threshold scoring
- Working knowledge of observability tools for language-model systems
- Ability to track quality over time, set baselines and detect regressions
- Regular use of agentic coding tools for multi-step tasks
- Clear technical writing
Qualifications
- Built or grew an evaluation practice across several teams
- Improved existing enterprise codebases with AI assistance
- Master's or PhD in ML, computing, applied maths or a closely related quantitative field
- Published articles, conference talks or open-source contributions in evaluation or LLMOps
Core Skills
Benefits
- Share options through the employee share option program
- Learning programs and coaching
- Quarterly refresh days, an extended year-end break and a monthly wellbeing allowance
- Parental leave from day one
- A MacBook and a budget for a home workspace
- Five paid social impact days each year
- Medical insurance for employee and family (US and UK only)
Frequently Asked Questions
Answers are based only on the employer’s listing; where it doesn’t say, neither do we.
Where is the role based?
The posting says the role is based in the Melbourne or Sydney hubs. Applicants need legal work authorisation in Australia throughout employment.
What is the main focus of the role?
The role centres on continuous production evaluation for the Coach AI system, including diagnosing performance shifts and driving quality gains.
Which tools are mentioned?
The posting mentions Langfuse for trace-based evaluation and observability, along with agentic coding tools such as Claude Code, Cursor and Codex.
Is a salary stated?
No. The posting does not state a salary.
Sample Interview Questions
AI-generated questions tailored to this specific role — a preview of the full practice set.
