Staff Software Engineer - Security Chaos Engineering

Staff Software Engineer - Security Chaos Engineering

Security Chaos Engineering is a group within our Resilience Engineering organization whose mission is to enable engineering teams at Datadog to improve and maintain the resilience of their services. We offer engineers platforms that they can use to verify if their systems operate as expected by embracing a mindset of experimentation and the practices of Chaos Engineering. Being able to simulate realistic failures in a complex ecosystem like Datadog, while limiting the blast radius in a safe way, requires our engineers to be strong software engineers with a good awareness of how distributed systems can break at scale.

As a Staff Engineer, you will help us realize this mission of enabling Chaos Engineering by continuing to build out our platforms and leveraging your influence by signal boosting the benefits of our work across the company. You will play an instrumental role in helping us build more adoption and impact on company-wide projects, and actively drive collaboration between the team and key stakeholders so that we build a culture of Chaos at Datadog.

What You’ll Do:

• Mentor engineers around the globe who are passionate about building a culture of reliability at Datadog. Help engineers of all seniority levels grow to the next level and continuously provide them opportunities to develop. 
• Lead projects to build exciting new features for our self-service platforms and ensure we are not only solving real user problems, but also ensure our platforms deliver real business impact to Datadog.
• Advocate for and build out Chaos Engineering as a practice across Datadog by leading through influence and empowering engineers at Datadog to verify their resilience to failures, including in production.
• Work with stakeholders across Datadog to build consensus on shared topics around resilience – for example collaborating with infrastructure teams to ensure we have consistent graceful degradation and disaster recovery solutions, or security teams to ensure adversary testing works for engineers.
• Build a culture of continuous learning by incentivizing engineers to invest in chaos experiments, team-level game days, and ways to proactively discover failures before they happen in production. 

Who You Are:

• Over 5 years of experience in the Chaos, Resilience or Reliability Engineering domain, with concrete experience leading transformational projects involving other engineering teams. Curiosity in bringing chaos engineering to different domains (such as security).
• 2-3 years experience serving in a Staff Engineer or leadership role in a large, complex organisation. We are looking for strong technical leadership skills and an ability to leverage your influence and relationship-building skills to drive impact.
• Experience with measuring Chaos Engineering programs and how they can be tied to business impact. An understanding of “measuring what matters” so that teams know the why behind our work. 
• Technical pragmatism and an ability to help the team reason about trade-offs around implementation. You will often review the decisions and RFCs from senior engineers, and you will need to blend both your technical and business acumen to do this. 
• Strong distributed systems knowledge, especially around Kubernetes and how controllers are designed, implemented and operated. We collaborate a lot with our infrastructure partners, so a solid understanding of Linux internals (e.g. control groups, namespaces and networking stack) will serve you well leading a technical team.
• Strong stakeholder management skills. This will involve using your empathy, collaboration, and communication skills in English to work remotely with people across teams. Because our programs touch lots of different parts of the business, we frequently collaborate with stakeholders across Datadog and need to motivate teams to work together towards a shared goal.