ML Platform Engineer, Distributed Training

Market insights

Stack

Apply

Upload your CV — we will connect you with the employer directly through our pool.

Description

Accountabilities - Optimize the training platform: Partner with infrastructure teams to improve GKE-based training environments, including GPU node pools, job scheduling, queueing, resource quotas, and capacity management. - Diagnose platform failures: Investigate underlying causes of infrastructure and training issues, resolve systemic problems, and implement durable fixes rather than relying on temporary workarounds. - Tune GPU networking: Optimize multi-node GPU communication and training throughput using technologies and techniques such as RDMA, GPUDirect, NCCL, and GPU interconnect topology. - Run distributed training workloads: Execute, scale, profile, and debug multi-node training jobs using PyTorch-based frameworks such as FSDP, DeepSpeed, Megatron, or NeMo, alongside Ray Train. - Identify performance bottlenecks: Analyze compute utilization, memory consumption, input pipelines, and network communication to improve training efficiency and scalability. - Design performance benchmarks: Develop and run benchmarks that measure throughput, scaling efficiency, and cost per training run, using the results to guide engineering decisions. - Build model evaluation systems: Create and maintain evaluation harnesses that objectively measure the impact of changes to training recipes, models, and platform configurations. - Improve observability and testing: Instrument training workloads and platform components with telemetry and monitoring, and maintain end-to-end pipeline tests tha

Employer contacts (email/phone/telegram) are hidden from the public preview — send your CV, and we will connect you directly.