Senior Software Engineer, Observability

About the role

The Observability department plays a pivotal role in CoreWeave’s ability to bring the best performing systems to the market. Whether you focus on metrics, logging and tracing, or pipelines and visualization, you’ll have an outsize opportunity to enable both CoreWeave and its customers to understand, troubleshoot, and optimize complex systems at the forefront of Artificial Intelligence.

We are seeking senior observability engineers with specializations in logging and tracing platforms (where knowledge of Loki and/or ClickHouse is a strong plus), pipelines and visualisation (where knowledge of Kafka and Grafana will set you apart). As you work closely with a tightly knit group of observability experts, you will have the opportunity to:

• Build low-touch, automated tooling that enables CoreWeave engineers to clearly articulate which pieces of telemetry, visualizations, and dashboards should be presented to customers.
• Modernize logging platforms at cloud-scale.
• Design migrations that are transparent to platform consumers.
• Assist CoreWeavers in differentiating between optimal use cases for structured and unstructured logs.
• Build governance mechanisms that empower CoreWeavers to effectively manage the telemetry their services produce and adopt best practices.
• Assist engineers across CoreWeave’s in maximizing the insights they glean from all three pillars of observability.
• Develop and enforce best practices regarding the health of telemetry ETL pipelines.
• Improve the performance, security, reliability, and scalability of observability services while participating in the team’s on-call rotation.
• Work with telemetry data at enormous scale, ingesting data from industry-leading GPU clusters.
• Grow, change, invest in your teammates, be invested-in, share your ideas, listen to others, be curious, have fun, and, above all, be yourself.

Wondering if you’re a good fit? We believe in investing in our people, and value candidates who can bring their own diversified experiences to our teams – even if you arent a 100% skill or experience match. Here are some qualities we’ve found compatible with our team. If a portion of this resonates with you, we’d love to talk. 

• You have six or more years of experience in a software or infrastructure engineering industry.
• You enjoy helping your colleagues achieve more with less effort.
• You are customer obsessed, ecstatic to provide infrastructure as a service, and default to adopting a product lens when evaluating platform scale problems.
• You have experience with logging and tracing platforms in production and at scale, and are versed in reliability engineering concepts such as the different types of testing, progressive deployments, error budgets, the role observability plays, and fault-tolerant design.
• You are opinionated about the best use cases for each of the three Observability pillars, and are excited about enabling engineers of all stripes - from frontend down to the kernel – with the best tools for their specific jobs.
• You have experience using Kubernetes, a conceptual understanding of its major components, and/or have operated Kubernetes clusters at scale for both event-driven and stateful orchestration.
• You’re familiar with various logging and metrics systems like ClickHouse, Elastic, Loki, Victoria Metrics, Prometheus, Thanos and Grafana. You have experience with designing and operating these systems at scale.
• You are familiar with PromQL, any other querying language and enjoy understanding the data model for observability systems. 
• You are excited at the prospect of zero-downtime, transparent migrations of major production services.
• You’re comfortable with the idea of using Go as your primary programming language, but don’t shy away from templating yaml as circumstances call for it.
• You know your way around a Linux distro, shell scripting, and/or the Linux storage and networking stacks.
• You can transform problems in elastic solutions, decompose them into achievable tasks, and socialize both to your teammates.
• You’re excited about being part of a team of diverse perspectives and backgrounds that believe in tackling challenges, growing hand in hand, and winning together.

Our compensation reflects the cost of labor across several US geographic markets. The base pay for this position ranges from $185,000-$210,000. Pay is based on a number of factors including market location and may vary depending on job-related knowledge, skills, and experience.