Remote opportunity at
EgenML Platform Engineer, Distributed Training
Egen seeks an ML Platform Engineer specializing in Distributed Training to join a team supporting a healthcare and life-sciences AI client. The organization operates with a…
Career Tools
About This Role
Egen seeks an ML Platform Engineer specializing in Distributed Training to join a team supporting a healthcare and life-sciences AI client. The organization operates with a data-centric perspective and brings engineering professionals together to apply advanced technologies like Google Cloud for enterprise clients. In this position, the professional will embed with a client team working on foundation models utilizing large GKE-based H200 and GB200 clusters. Rather than managing foundational cluster…
Job Description
Egen seeks an ML Platform Engineer specializing in Distributed Training to join a team supporting a healthcare and life-sciences AI client. The organization operates with a data-centric perspective and brings engineering professionals together to apply advanced technologies like Google Cloud for enterprise clients. In this position, the professional will embed with a client team working on foundation models utilizing large GKE-based H200 and GB200 clusters. Rather than managing foundational cluster infrastructure directly, the individual will collaborate closely with dedicated site reliability and infrastructure teams to tune, benchmark, and troubleshoot distributed training workloads comprehensively.
Core duties involve optimizing GPU networking protocols such as NCCL and RDMA, scaling multi-node distributed training routines through PyTorch and Ray Train, conducting rigorous performance benchmarks, and creating evaluation harnesses. Candidates must demonstrate deep technical proficiency with Google Cloud, Kubernetes GPU node pools, Python programming, and hardware-level performance optimization. The engagement requires autonomous ownership, strong communication skills to defend technical decisions, and the ability to adapt as client priorities evolve. Total yearly compensation ranges between $159,300 and $219,245, with final leveling established during interviews based on background and expertise.
Responsibilities
- Collaborate with the client infrastructure group to support GKE training platforms, managing GPU node pools, capacity, quotas, and job scheduling
- Identify and resolve deep platform defects at their source rather than merely restarting failed processes
- Optimize multi-node GPU interconnects and networking topologies including RDMA, GPUDirect, and NCCL
- Execute, scale, and troubleshoot multi-node distributed training pipelines utilizing Ray Train and PyTorch frameworks
- Profile execution jobs to eliminate performance bottlenecks across hardware compute, memory tiers, and input data streams
- Design and execute performance measurements covering scaling efficiency, run costs, and raw throughput
- Construct and maintain evaluation frameworks to measure the impact of platform adjustments and training recipe updates
- Implement monitoring and telemetry solutions to maintain active pipeline testing against user workflows
Requirements
- Minimum of 5 years of engineering experience featuring recent, hands-on involvement with large-scale training and GPU training infrastructure
- Advanced Python software development capabilities, producing production-grade code, conducting reviews, and writing tests
- Practical expertise with Google Cloud Kubernetes environments featuring GPU node pools, resource quota administration, and job queueing tools
- Demonstrated history of personally operating and debugging multi-node distributed training jobs using PyTorch and Ray Train
- Familiarity with GPU networking configurations, NCCL tuning, and multi-node H200 or GB200 hardware architectures
- Experience formulating performance benchmarks and model evaluation frameworks to drive technical decisions
- Demonstrated accountability, autonomy, and professional communication skills on high-demand teams
Qualifications
- Background handling healthcare or life-sciences data sets such as genomics, clinical information, or biomedical records under HIPAA guidelines
- Experience implementing observability, metrics, tracing, alerts, and end-to-end testing for machine learning training environments
Core Skills
Benefits
- Comprehensive health insurance coverage
- Paid leave options including vacation and paid time off
- Paid holidays
- Sick leave policies
- Parental leave support
- Bereavement leave
- Employer matched 401(k) retirement plan
- Employee referral incentive bonuses
Frequently Asked Questions
Answers are based only on the employer’s listing; where it doesn’t say, neither do we.
Is this position remote?
Yes, the posting explicitly lists the remote status as remote.
What is the salary range for this role?
The annual salary ranges from $159,300 to $219,245, with final placement determined by qualifications and interview outcomes.
What level of experience is required?
Applicants must have at least 5 years of engineering experience with recent hands-on work in large-scale training and GPU training infrastructure.
Sample Interview Questions
AI-generated questions tailored to this specific role — a preview of the full practice set.