Staff Software Engineer, Speculative Decoding

Staff Software Engineer, Speculative Decoding

Mission: with a strong background in Generative AI Inference and expertise in Speculative Decoding, you will design, implement, and optimize cutting-edge algorithms to enhance our production AI infrastructure and capabilities in post training, model evaluation, and operational performance.

Responsibilities & outcomes:

• Design, implement, and optimize speculative decoding algorithms and underlying models that enhance the speed and accuracy of Generative AI Inference.
• Collaborate with cross-functional teams to integrate your solutions into Groq’s production AI infrastructure. 
• Work in a multi data center production environment and Kubernetes environment with Groq’s customer hardware, inference and compiler stack.
• Develop high-performance, scalable code primarily in C++ and Rust, ensuring efficient resource utilization and system stability. Ability to model performance of a distributed high performance system.
• Experience building production distributed systems involving multi process communication with technologies such as MPI, scheduling and working in a kubernetes environment.
• Stay up-to-date with the latest developments in generative AI and speculative decoding, and translate cutting-edge research into practical, production-ready implementations.
• Work closely with teams across software engineering, research, and operations to drive improvements in post training, model evaluation, and overall system performance.
• Provide technical leadership and mentorship to team members, fostering an environment of continuous learning and innovation.
• Champion code quality, maintainability, observability, monitoring and best practices, ensuring that all deliverables meet rigorous performance and security standards.

Ideal candidates have/are:

• Master’s degree in Computer Science, Electrical Engineering, or a related field (or equivalent industry experience).
• Extensive, hands-on experience in generative AI inference with a specific focus on speculative decoding.
• Proficiency in C++ is essential, with demonstrated experience in developing high-performance systems.
• Strong analytical and problem-solving skills, with a track record of delivering innovative technical solutions.
• Proven ability to work effectively in fast-paced, cross-functional environments, driving projects from conception to production.
• Understanding of the architecture of Generative AI models, PyTorch, familiarity with the data science necessary to evaluate layers of models, their performance and quality.
• Familiarity with AI infrastructure challenges and scalable system design.

Attributes of a Groqster:

• Humility - Egos are checked at the door
• Collaborative & Team Savvy - We make up the smartest person in the room, together
• Growth & Giver Mindset - Learn it all versus know it all, we share knowledge generously
• Curious & Innovative - Take a creative approach to projects, problems, and design
• Passion, Grit, & Boldness - no limit thinking, fueling informed risk taking

If this sounds like you, we’d love to hear from you!

Compensation: At Groq, a competitive base salary is part of our comprehensive compensation package, which includes equity and benefits. For this role, the base salary range is $175,900 to $307,800, determined by your skills, qualifications, experience and internal benchmarks.