Helix AI Engineer, Training Performance
FigureSan Jose, California, United States · Posted 14 days agoDescription
Figure's vision is to deploy autonomous humanoids at a global scale. Our Helix team is looking for an experienced AI Training Performance Engineer to take our model training to the next level. This role is focused on improving distributed training frameworks for large scale model training, optimizing GPU kernels, exploring the relative gains of different accelerator types and co-designing our models to maximize utilization of our hardware.
Responsibilities
- Optimize training performance for a 100B+ parameter models across 100k+ GPUs.
- Collaborate with the broader team on accelerator choice, cluster topology, scheduling, and hardware procurement decisions to inform future scaling.
- Write and optimize custom kernels (Triton/CUDA)
- Build tooling and dashboards for continuous performance monitoring, regression detection, and root-cause analysis across training jobs
- Optimize data loading and preprocessing pipelines so I/O never gates the accelerators
- Improve checkpointing, fault tolerance, and elastic restart so large jobs recover quickly from node failures without losing significant wall-clock time
- Partner with researchers to co-design model architectures and training recipes that are performant at scale (e.g., activation checkpointing strategies, mixed precision, sequence packing)
- Extend and contribute to kernel compilers (e.g., Triton, Gluon) to improve iteration speed and enable targeting of custom/non-NVIDIA accelerators
- Build and extend agentic systems that automatically generate, benchmark, and iterate on custom kernels
- Evaluate emerging accelerator architectures (AMD, TPU, SRAM-based ASICs, and other novel hardware) for fit with our training workloads, and lead proof-of-concept ports/benchmarks
- Explore different model/data parallelisms (FSDP, context parallel, expert parallel, etc.) to determine optimal configuration per model size.
Requirements
- Bachelor's or Master's degree in Computer Science, Computer/Electrical Engineering, or a related field
- 3+ years in AI performance engineering, with significant time leading large-scale performance improvement projects
- Deep understanding of GPU architecture and performance characteristics (memory bandwidth, compute-bound vs. memory-bound ops, occupancy)
- Proficiency with profiling tools (Nsight Systems/Compute, PyTorch Profiler, HTA, or similar) and ability to translate traces into concrete optimizations
- Solid grasp of collective communication (NCCL) and modern networking concepts (RDMA, NVLink, InfiniBand/RoCE, topology-aware placement).
- Strong Python and CUDA/C++ skills; comfortable reading and modifying framework internals
- Experience debugging performance regressions and instability at scale (stragglers, hangs, OOMs, numerical divergence)
- Experience defining and reasoning about hardware-efficiency metrics (MFU/HFU) and using them to drive optimization priorities
Bonus Qualifications
- Experience with heterogeneous or multi-datacenter training setups and cross-cluster orchestration
- Contributions to open-source ML systems projects (PyTorch, Megatron-LM, vLLM, DeepSpeed, JAX, etc.)
- Exposure to non-NVIDIA accelerators (AMD GPUs, TPU/Trainium/Inferentia, or custom silicon) and heterogeneous fleet management.
The US base salary range for this full-time position is between $200,000 - $400,000 annually.
The pay offered for this position may vary based on several individual factors, including job-related knowledge, skills, and experience. The total compensation package may also include additional components/benefits depending on the specific role. This information will be shared if an employment offer is extended.
Similar jobs
Est. 300,000 USD
Figure is an AI robotics company developing autonomous general-purpose humanoid robots. Our goal is to build embodied AI systems that can perceive, reason, and act in the real world. Figure is headquartered in San Jose,…
Est. 275,000 USD
Figure is an AI Robotics company developing a general purpose humanoid. Our humanoid robot is designed for commercial tasks and the home. We are based in San Jose and require 5 days/week in-office collaboration. It’s tim…
Est. 227,500 USD
Figure is an AI robotics company developing autonomous general-purpose humanoid robots. The goal of the company is to ship humanoid robots with human level intelligence. Its robots are engineered to perform a variety of…
Est. 300,000 USD
Figure is an AI robotics company developing autonomous general-purpose humanoid robots. Our goal is to build embodied AI systems that can perceive, reason, and act in the real world. Figure is headquartered in San Jose,…
Est. 300,000 USD
Figure is an AI robotics company developing autonomous general-purpose humanoid robots. Our goal is to build embodied AI systems that can perceive, reason, and act in the real world. Figure is headquartered in San Jose,…
Est. 300,000 USD
Figure is an AI robotics company developing autonomous general-purpose humanoid robots. Our goal is to build embodied AI systems that can perceive, reason, and act in the real world. Figure is headquartered in San Jose,…
Est. 300,000 USD
Figure is an AI robotics company developing autonomous general-purpose humanoid robots. Our goal is to build embodied AI systems that can perceive, reason, and act in the real world. Figure is headquartered in San Jose,…
Est. 300,000 USD
Figure is an AI Robotics company developing a general purpose humanoid. Our humanoid robot, Figure 03, is designed for commercial tasks and the home. We are based in San Jose, CA and require 5 days/week in-office collabo…
Est. 300,000 USD
Figure is an AI robotics company developing autonomous general-purpose humanoid robots. The goal of the company is to ship humanoid robots with human level intelligence. Its robots are engineered to perform a variety of…
Est. 300,000 USD
Figure is an AI Robotics company developing a general purpose humanoid. Our humanoid robot is designed for commercial tasks and the home. We are based in San Jose and require 5 days/week in-office collaboration. It’s tim…
Est. 300,000 USD
Figure is on a mission to develop and deploy general purpose humanoid robots for corporate tasks targeting labor shortages and jobs that are undesirable or unsafe. We are looking for passionate and motivated superstars t…
Est. 225,000 USD
Figure is an AI Robotics company autonomous general-purpose humanoid robots. The goal of the company is to ship humanoid robots with human level intelligence. Its robots are engineered to perform a variety of tasks in th…
Est. 62,400 USD
Figure is an AI robotics company developing autonomous general-purpose humanoid robots. The goal of the company is to ship humanoid robots with human level intelligence. Its robots are engineered to perform a variety of…
Est. 144,000 USD
Figure is an AI Robotics company developing a general purpose humanoid. Our humanoid robot, Figure 03, is designed for commercial tasks and the home. We are based in San Jose, CA and require 5 days/week in-office collabo…
Est. 62,400 USD
Figure is an AI robotics company developing autonomous general-purpose humanoid robots. The goal of the company is to ship humanoid robots with human level intelligence. Its robots are engineered to perform a variety of…
Est. 212,500 USD
Figure is an AI robotics company developing autonomous general-purpose humanoid robots. The goal of the company is to ship humanoid robots with human level intelligence. Its robots are engineered to perform a variety of…
Est. 160,000 USD
Figure is an AI robotics company developing autonomous general-purpose humanoid robots. The goal of the company is to ship humanoid robots with human level intelligence. Its robots are engineered to perform a variety of…
Est. 300,000 USD
Figure is an AI Robotics company developing a general purpose humanoid. Our humanoid robot, Figure 03, is designed for commercial tasks and the home. We are based in San Jose, CA and require 5 days/week in-office collabo…
Est. 315,000 USD
About Hark Hark is an artificial intelligence company building advanced, personalized intelligence. One that is proactive, multimodal, and capable of interacting with the world through speech, text, vision, and persisten…
Est. 315,000 USD
About Hark Hark is an artificial intelligence company building advanced, personalized intelligence. One that is proactive, multimodal, and capable of interacting with the world through speech, text, vision, and persisten…
Est. 130,000 USD
Figure is an AI Robotics company developing autonomous general-purpose humanoid robots. The goal of the company is to ship humanoid robots with human level intelligence. Its robots are engineered to perform a variety of…
Est. 67,200 USD
Figure is an AI robotics company developing autonomous general-purpose humanoid robots. The goal of the company is to ship humanoid robots with human level intelligence. Its robots are engineered to perform a variety of…
Est. 205,000 USD
Figure is an AI Robotics company developing a general purpose humanoid. Our humanoid robot, Figure 02, is designed for commercial tasks and the home. We are based in San Jose, CA and require 5 days/week in-office collabo…
Est. 210,000 USD
About Hark Hark is an artificial intelligence company building advanced, personalized intelligence. One that is proactive, multimodal, and capable of interacting with the world through speech, text, vision, and persisten…
Est. 212,500 USD
Figure is an AI robotics company developing autonomous general-purpose humanoid robots. The goal of the company is to ship humanoid robots with human level intelligence. Its robots are engineered to perform a variety of…
Est. 185,000 USD
Figure is an AI robotics company developing autonomous general-purpose humanoid robots. The goal of the company is to ship humanoid robots with human level intelligence. Its robots are engineered to perform a variety of…
Est. 83,200 USD
Figure is an AI Robotics company developing autonomous general-purpose humanoid robots. The goal of the company is to ship humanoid robots with human level intelligence. Its robots are engineered to perform a variety of…
Est. 300,000 USD
Jump Trading Group is committed to world class research. We empower exceptional talents in Mathematics, Physics, and Computer Science to seek scientific boundaries, push through them, and apply cutting edge research to g…
Est. 185,000 USD
Figure is an AI robotics company developing autonomous general-purpose humanoid robots. The goal of the company is to ship humanoid robots with human level intelligence. Its robots are engineered to perform a variety of…
Est. 262,500 USD
Figure is an AI robotics company developing autonomous general-purpose humanoid robots. The goal of the company is to ship humanoid robots with human-level intelligence. Our robots are engineered to perform a variety of…