Job Description, Responsibilities & Requirements
About the Position
We are seeking a Principal HPC Network Engineer to support advanced AI, research, and Kubernetes-based GPU infrastructure for a major global technology client.
An ideal candidate is an ownership-driven, hands-on network expert who combines deep low-level troubleshooting with architectural thinking and clear communication in complex AI/HPC, Kubernetes, and GPU infrastructure environments.
Responsibilities
- Architect, operate, and troubleshoot high-performance InfiniBand/RDMA and Ethernet fabrics for large-scale GPU clusters and distributed AI/LLM workloads
- Design and evaluate cluster network topologies, including Fat-tree, Clos, Rail-optimized, and Dragonfly, based on workload scale and performance needs
- Optimization of host-side networking, including NIC configuration, drivers, firmware, IRQ affinity, NUMA placement, PCIe topology, and GPU-to-NIC communication paths
- Tune and troubleshoot RDMA/RoCE, NCCL/MSCCL, and collective communication performance for multi-node GPU training workloads
- Design and maintain Kubernetes networking for GPU clusters, including CNI plugins, network policies, multi-NIC pods, RDMA/GPU device plugins, and workload orchestration integration
- Support for SmartNIC/DPU technologies such as NVIDIA BlueField where applicable, including SR-IOV, offload, isolation, and security use cases
- Build and improve network observability, including metrics, dashboards, alerts, congestion detection, latency tracing, SLO reporting, and capacity/performance analysis
- Collaboration with Kubernetes, storage, GPU infrastructure, observability, and AI research teams to resolve network and I/O bottlenecks and improve workload reliability
Requirements
- 8+ years of experience in network, infrastructure, HPC, SRE, or similar engineering roles, with 4+ years focused on HPC, AI/ML, or GPU cluster networking
- Expert-level experience in high-performance networking for HPC, AI/ML, GPU clusters, or large-scale compute environments
- Knowledge of InfiniBand fabrics, including NDR/HDR or comparable high-speed generations, subnet managers, fabric configuration, topology, and troubleshooting
- Understanding of RDMA networking concepts, including InfiniBand, RoCE/RoCEv2, GPUDirect-related patterns, congestion behavior, and performance tuning
- Skills in Kubernetes and container networking for GPU or distributed workloads, including CNI concepts, network policies, multi-NIC patterns, and RDMA/GPU device integration
- Proficiency in Linux networking and host-side troubleshooting, including NIC configuration, drivers, firmware, IRQ affinity, NUMA awareness, PCIe topology, MTU, offloads, and performance diagnostics
- Expertise in network observability and performance management, including telemetry, traffic monitoring, congestion detection, latency analysis, SLOs, capacity planning, alerting, and troubleshooting across L1-L4, fabric, and RDMA layers
- Practical knowledge of distributed AI training communication patterns, including NCCL-based workloads and collective operations such as all-reduce and all-gather
- Familiarity with host-side networking, including NICs, PCIe topology, NUMA awareness, and GPU-to-NIC affinity
- Capability to perform troubleshooting, root-cause analysis, documentation, and communication with client engineering teams, researchers, and platform stakeholders
Nice to Have
- Knowledge of Azure Networking, Ethernet, and GPGPU/GPU
- Skills in Grafana, Prometheus, and Network Administration
- Proficiency in Python and UNIX shell scripting
- Capability to perform Infrastructure as Code development and maintenance
About the Company
[Company description if present]