Description
Role Summary
The AI Engineer will establish Firmus AI Factory as the foundation for efficient, production-grade distributed training by delivering pre-built training recipes (TorchTitan, Megatron etc.), evaluation benchmarks, and model guidance. You'll work with customers and internal teams to optimize training efficiency, define baselines, and document best practices. Your templates and benchmarks are the anchor point for our hyperscale customers' training workflows and our model arena differentiator.
Key Responsibilities
- Build production-ready training recipes using TorchTitan and Megatron-LM: model configs, parallelism strategies (FSDP, tensor/pipeline parallelism), checkpointing patterns.
- Document parameter tuning for different scales (e.g., "to train Llama 7B on 8xH100s, use this config and expect X throughput").
- Create and validate multi-node NCCL communication patterns on AI Factory K8s/Slurm clusters.
- Design and build benchmarking suites: accuracy, latency, throughput (tokens/sec), cost per token, energy efficiency, MFU.
- Implement offline evaluation harnesses for standardized model comparison and leaderboard tracking.
- Conduct fine-tuning experiments (LoRA, QLoRA) where they improve product outcomes (e.g., ops domain data), document gains.
- Create training efficiency playbooks and publish benchmark results so customers can optimize workloads.
- Partner with job scheduling and orchestration engineers on template integration and other AI engineers and software engineers on model optimization trade-offs for inferencing and AI applications.
Skills & Experience
- 5–7 years of experience in distributed machine learning (PyTorch/JAX, FSDP, DeepSpeed, multi-node training at 10+ GPUs).
- Expert-level understanding of GPU optimization: utilization, memory patterns, communication bottlenecks (NCCL collectives).
- Hands-on distributed training at scale: debugged convergence issues, profiled bottlenecks, optimized throughput.
- Strong benchmarking methodology: design-controlled experiments, measure noise, communicate results rigorously.
- Familiarity with TorchTitan, Megatron-LM, or similar production training frameworks.
- Understanding of model parallelism strategies and trade-offs (FSDP vs. tensor parallelism vs. pipeline parallelism etc.).
Key Competencies
- Distributed Systems Mastery: can explain NCCL, collective communications, and scaling inefficiency.
- Benchmarking Rigor: doesn't just run benchmarks; validates assumptions, explains variance, communicates uncertainty.
- Production Thinking: understands checkpointing, recovery, resource constraints, and cost optimization.
- Mentorship: can guide engineers on training best practices and debugging distributed training issues.
- Documentation: creates clear, actionable playbooks that customers can follow.
Success Metrics
- Benchmark credibility & decision impact increases: benchmarks are trusted and used to drive model/hardware/product decisions.
- Training efficiency leadership: sustained improvement in benchmarked training efficiency on representative workloads.
- Shorter time-to-validate new models: model candidates can be evaluated quickly and consistently end-to-end.
- Template effectiveness improves: recipes reduce misconfigurations and repeated setup failures; fewer training config escalations.
- Competitive differentiation strengthens: model arena outputs influence customer adoption and internal roadmap priorities.
Location & Reporting
- Singapore or Australia (Launceston, TAS or Sydney, NSW)
- Reporting to Head of AI & Applications
Employment Basis
Full-time
Diversity
At Firmus, we are committed to building a diverse and inclusive workplace. We encourage applications from candidates of all backgrounds who are passionate about creating a more sustainable future through innovative engineering solutions.
Join us in our mission to revolutionize the AI industry through sustainable practices and cutting-edge engineering. Apply now to be part of shaping the future of sustainable AI infrastructure.
Similar jobs
Role Summary Every AI feature we ship touches thousands of GPUs. The Senior DevOps Engineer will build the release engineering backbone—CI/CD pipelines, automated testing gates, one-click deployments with instant rollbac…
Firmus Technologies Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific. Founded in Australia in 2019, our mission is to create the most effi…
Role Summary Firmus is seeking a highly skilled and driven Senior Engineer to play a key role in designing, building, and operating software-defined infrastructure, including high-performance AI storage platforms. You wi…
Firmus Technology Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific. Founded in Australia in 2019, our mission is to create the most effici…
Est. 165,000 USD
Firmus Technologies Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure from model to grid. Founded in Australia in 2019, our mission is to create the most effic…
Firmus Technologies Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific. Founded in Australia in 2019, our mission is to create the most effi…
Firmus Technologies Firmus Technologies is a global leader pioneering the solution to AI’s energy challenge, founded in Australia in 2019 by a visionary team of entrepreneurs. Our mission is to create the most energy-eff…
Firmus Technologies Firmus Technologies is a global leader pioneering the solution to AI’s energy challenge, founded in Australia in 2019 by a visionary team of entrepreneurs. Our mission is to create the most energy-eff…
Est. 165,000 USD
Firmus Technologies Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific. Founded in Australia in 2019, our mission is to create the most effi…
Firmus Technologies Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific. Founded in Australia in 2019, our mission is to create the most effi…
Firmus Technologies Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific. Founded in Australia in 2019, our mission is to create the most effi…
Firmus Technologies Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific. Founded in Australia in 2019, our mission is to create the most effi…
Firmus Technologies Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific. Founded in Australia in 2019, our mission is to create the most effi…
Firmus Technologies Firmus Technologies is a global leader pioneering the solution to AI’s energy challenge, founded in Australia in 2019 by a visionary team of entrepreneurs. Our mission is to create the most energy-eff…
Est. 250,000 USD
Firmus Technologies Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure from model to grid. Founded in Australia in 2019, our mission is to create the most effic…
Firmus Technologies Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific. Founded in Australia in 2019, our mission is to create the most effi…
Firmus Technologies Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific. Founded in Australia in 2019, our mission is to create the most effi…
Firmus Technologies Firmus Technologies is a global leader pioneering the solution to AI’s energy challenge, founded in Australia in 2019 by a visionary team of entrepreneurs. Our mission is to create the most energy-eff…
Firmus Technologies Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific. Founded in Australia in 2019, our mission is to create the most effi…
Firmus Technologies Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific. Founded in Australia in 2019, our mission is to create the most effi…
Firmus Technologies Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific. Founded in Australia in 2019, our mission is to create the most effi…
Firmus Technologies Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific. Founded in Australia in 2019, our mission is to create the most effi…
Firmus Technologies Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific. Founded in Australia in 2019, our mission is to create the most effi…
Firmus Technologies Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure from model to grid. Founded in Australia in 2019, our mission is to create the most effic…
Firmus Technologies Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific. Founded in Australia in 2019, our mission is to create the most effi…
Firmus Technologies Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific. Founded in Australia in 2019, our mission is to create the most effi…
About the role As a Forward Deployed Engineer (FDE) focused on Inference & Post-Training, you will be a hands-on technical partner to our most strategic customers — production AI teams looking to leverage high qualit…
Firmus Technologies Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific. Founded in Australia in 2019, our mission is to create the most effi…
Firmus Technologies Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific. Founded in Australia in 2019, our mission is to create the most effi…
Est. 205,000 USD
About Turing Turing’s mission is to accelerate superintelligence to drive real economic progress. Headquartered in San Francisco, Turing works with frontier AI labs to generate high-quality datasets, reinforcement learni…