Staff AI Product Engineer

About the Position

Nscale is seeking a Staff AI Product Engineer to lead technical direction for our AI cloud inference platform, focusing on performance and reliability. This role is pivotal in shaping the latency, throughput, and cost of the tokens we serve, ensuring the models we run are the right ones.

About the Role

Nscale is looking for a Staff AI Engineer (Specialised) to set technical direction for part of the inference platform at the core of our AI cloud: dedicated and serverless inference, bring-your-own-model deployments, and the post-training services that sit alongside them.

You’ll be the technical authority for one or more focus areas, working across the teams that build serving, post-training, and platform. Your decisions shape the latency, throughput, and cost of the tokens Nscale serves, and whether we can say with evidence that the models we run are the right ones. You’ll take on open questions, such as how KV cache should move between GPUs, nodes, and storage, or where hand-written kernels beat the open-source defaults, and your answers become the standards others build on.

Responsibilities

• Set technical direction for your focus areas and turn it into work that multiple teams can deliver
• Lead the resolution of systemic performance and reliability problems across the serving stack, from kernel bottlenecks to fleet-level capacity and multi-tenant isolation
• Own the trade-offs between cost, latency, throughput, and model quality in your area, and back them with measurement
• Build reusable frameworks, benchmarks, and tooling that make other AI engineers at Nscale more effective
• Evaluate emerging serving engines, kernel libraries, RL frameworks, and accelerators, and make clear build/adopt/contribute recommendations
• Coach and grow engineers across teams, and raise engineering quality broadly
• Work with research, product, and infrastructure leadership so the platform tracks customer demand
• Represent your area in cross-team technical reviews and planning

Requirements

• 8–12 years of engineering experience, with significant depth in production AI systems or ML research at scale
• 4+ years of hands-on work with LLMs in at least one of the focus areas above, in production or research
• Deep, demonstrated expertise in at least one of the focus areas above
• Enough working knowledge of the full serving stack to debug across it: API, router, scheduler, engine, kernel, and cluster
• A track record of setting technical direction and creating standards or tools adopted beyond your own team
• Strong Python and PyTorch, with a track record of maintainable, well-tested, production-grade systems
• Experience operating AI workloads in containerised, distributed environments (Kubernetes, large GPU clusters)
• Solid understanding of GPU performance: memory bandwidth, interconnect, and parallelism strategies (data, tensor, pipeline, expert)
• Deep understanding of transformer and LLM architectures and how they behave under production load

Preferred

• Contributions to widely used open-source inference, kernel, or RL projects (vLLM, SGLang, TensorRT-LLM, LMCache, FlashInfer, Triton, verl, OpenRLHF, TRL, DeepSpeed, etc.)
• Experience at an AI lab, hyperscaler AI team, or leading ML infrastructure company
• Published research or technical writing on inference, kernels, evals, or RL (MLSys, NeurIPS, ICLR, blog posts)
• Hands-on custom kernel work (CUDA, Triton, CUTLASS) if kernels aren’t already your focus area
• Experience designing developer-facing APIs and SDKs used by external customers
• Experience with control plane / data plane separation and cell-based architecture for scale-out and blast-radius isolation

We Offer

• Competitive Salary: $220,000 - $330,000 USD
• Benefits: Medical, dental, vision, flexible paid time off, parental leave, and retirement plan participation
• Locations: Houston, New York, San Francisco, Seattle
• Work Type: Hybrid

About the Company

Nscale is taking on the hyperscalers by building a vertically integrated GenAI cloud platform. We own the data centers, software, and applications that power today's AI stack using sustainable technology solutions. We thrive on a culture of relentless innovation, ownership, and accountability, where every team member takes pride in their work and drives it with excellence and urgency. As a Nscaler, you'll build trust through openness and transparency, where everyone is inspired to do their best work. Collaboration is key, and we work together swiftly and respectfully, embracing adaptability and resilience in all we do.