6 posts tagged with "sre"

An SRE lead at a mid-size fintech told me the quote that defines 2026 observability decisions: "Datadog is the iPhone of observability — expensive, polished, and I wish I had a choice." The market has three credible positions now: Datadog as the integrated default, Grafana as the open-source-first alternative, and Honeycomb as the wide-events specialist. Each is optimized for a different failure mode, and picking the wrong one doesn't show up in the first quarter — it shows up as a $2M annual bill and a team that still can't answer "why was latency spiky on Tuesday?"

CNCF's 2024 Annual Survey reported that 86% of cloud-native organizations use OpenTelemetry in some form — which sounds like the market is standardizing. In practice OTel is a pipeline, not a destination; every shop running it still picks one of these three stacks (or Splunk, New Relic, Dynatrace — we'll touch those briefly) to actually store, query, and visualize the data. Honeycomb's own observability maturity research shows that teams adopting wide-events cut investigation time on novel incidents by 40-60%, but only when the culture adapts — tooling alone doesn't deliver the lift.

The observability market crossed $5 billion in annual revenue in 2025 and is on track for another double-digit growth year in 2026. Two of the loudest names, Datadog and Honeycomb, sit at opposite philosophical poles. Datadog wants to be the single pane of glass for everything that breathes in your cluster. Honeycomb argues that "everything" is a trap, and that a single wide event per request beats three pillars stitched together with correlation IDs. Both are right about something. Neither is right about everything.

Two production outages, same root cause: a bad config push that crashed a payments service. Team A spent 2 hours 14 minutes restoring service. Team B was back in 6 minutes. Team B's MTTR wasn't lower because they had smarter engineers. They had a one-command rollback rehearsed monthly, a runbook pinned in the on-call channel, and write access to production already granted to the responder. That 134-minute gap is what MTTR measures, and what separates the DORA 2023 State of DevOps Report elite cluster from everyone else.

Your best SRE quit last quarter. She didn't say "burnout" in the exit interview, but her last three months included 14 after-hours pages, 2 weekend incidents, and a 3am call on her birthday. A 2021 Catchpoint / DevOps Institute survey of 500+ on-call engineers found 67% reported burnout symptoms tied directly to paging load. Google's SRE book sets an internal ceiling of 2 incidents per on-call shift before a rotation is declared unhealthy — most teams we measure blow past that in week one.

On-call is fixable. It's a scheduling and sociotechnical problem, not a personality flaw in the people who can't hack it. Here's a 9-rule playbook that keeps your SLA intact and keeps your best engineers on the team past their second rotation.

The average post-mortem takes 4 hours to write and generates zero action items the team actually completes within 30 days. We looked at 120 post-mortem documents from three of our on-prem customers before rebuilding this template. 83% of action items were still "open" six months later. That's not an incident review — that's a document graveyard.

A post-mortem is worth writing only if it changes something. Everything else is CYA.

Google's Site Reliability Engineering book (2016) popularized a counterintuitive principle: accept failure as inevitable and invest in recovery speed. The DORA research confirmed it with data — the difference between elite and low-performing teams isn't that elite teams have fewer incidents. It's that they recover in under an hour instead of under a week. Every engineering organization invests in preventing failures. Fewer invest in recovering from them quickly. The data says this is backwards.