“We don’t know how this outage affected our customers.”
An engineer said it at the end of a long postmortem, and nobody in the room could argue. We were running at two nines. Customers were losing trust in us. The on-call team was getting paged most nights.
Eighteen months later, usage had tripled. Reliability on our control plane had climbed to three nines, then four and a half, by our internal SLOs. A customer told us our software felt battle-tested. None of it was a solo act. It took every team that shipped to production, and a set of leaders who made it their own.
We climbed a ladder, in order: see the problem, make the work predictable, then hold people accountable for it.
Why reliability stalls
Reliability is a horizontal problem. Every team touches it and no team owns it. It loses every prioritization fight to the feature with a name and a launch date. A heroic reliability push fades the quarter after the hero moves on. Handing it to a central team makes it worse, because teams stop owning what they ship, and trust erodes.
So the fix starts above engineering. The CTO and the leaders across product and engineering have to agree, out loud, that reliability is part of the product. Ours was one sentence. We’re in the business of selling trust, and reliability and security are table-stakes features. Without that agreement, reliability work gets traded for a roadmap item at the first planning review.
A concern I kept hearing in skip-levels: ‘We can’t tell what’s running in production, and staging looks nothing like prod.’ In planning, we deferred a new platform investment to fund this work instead.
Once that sponsorship was in place, we climbed three rungs:
- Observability: see what’s breaking and who it hurts.
- Predictability: give teams the skills, tools and time to fix it.
- Accountability: a top-down cadence that makes sure the fixes land.
Each rung makes the next one possible, and fair. You can’t hold a team accountable for an SLO nobody measures, or for improvements they were never given time to make.
Rung 1 · Observability: see it
Our dashboards told us when systems were unhealthy. They couldn’t tell us which customers felt it. That’s what the engineer in the postmortem was saying.
We added three kinds of signals.
Customers. Support tickets and our customer advisory board told us where customers actually felt the pain.
Usage. Which parts of the product were used, and by whom. A flaky endpoint nobody calls is a different priority from one that your largest customers hit every second.
Ownership. Which of those heavily used parts had no owner at all. We were routinely breaching page-load SLOs on one of the most-used parts of our control plane, and found that it had no clear owner.
The first honest picture was ugly. We had too few SLOs. The ones we had measured our systems rather than what customers felt, and 70% of those were in breach. In one quarter, 14 of our 16 high-severity incidents were self-inflicted. And for nearly all, we lacked a high-confidence signal into customer impact.
The 14 of 16 reframed everything. Most of the pain came from our own changes and our own gaps, which meant it was ours to fix.
Rung 2 · Predictability: make it boring (bottom-up)
Predictability is change driven bottom-up. The engineers closest to the systems already know what’s fragile. They need the time, the tools and the support to fix it.
Time came first. We carved out run-the-business capacity in budgeting and headcount planning, so reliability work had its own line item instead of borrowing from features. We also reserved 10% of support engineering time for operational improvements.
Then people. We backed our engineers who cared about operational work, giving them the mandate and the time to lead it. We hired distributed-systems engineers and leadership from AWS and companies that had seen scale and rigor, people who had run infrastructure where reliability was the product. And we got deliberate about fit. Some engineers are scalers, who love hardening a system. Others are explorers, who thrive on zero-to-one. Put an explorer on ops rigor and they burn out. Put a scaler on demos and you waste them. We put each on the work they’re built for and enjoy doing.
Then tooling. We moved incidents and postmortems onto incident.io so the process lived in one place instead of in ad-hoc Slack threads and Confluence pages. This helped us mobilize for high-urgency incidents, roll out incident severity criteria and get methodical about tracking critical remediations from postmortems.
Predictability had a customer-facing half too. Our runtime layer, the Router, had no clear support policy, so every customer upgrade was a negotiation. We introduced semantic versioning. Then we agreed with customers how far back we’d support major and minor releases, which releases would get patches, and how we’d handle security vulnerabilities. Customers could plan their upgrades, and so could we.
Investing in a runtime testing framework cut Router release times from months to days.
Rung 3 · Accountability: own it (top-down)
Accountability is change driven top-down. Ours centered on a weekly Operational Excellence Review (OER).
OER wasn’t our first try. An earlier operational review, run by two managers who cared about this work, kept losing to everything else on teams’ plates. What changed was who showed up. When directors and I came every week and asked questions, teams knew it mattered.
We started with five audits:
- postmortems ready for review
- overdue action items
- high-urgency pages
- weekly SLAs
- cost per query
That list grew to 17 audits without the weekly 60-minute meeting getting any longer. Every audit was automated, a dashboard that showed green or red at a glance, because manual checks don’t scale over time. Operational excellence serves both the top line and the bottom line, so Finance and Customer Support were in the room too. Finance kept us honest on cost per query, and Support brought what customers were feeling that week.
One rule made it work: OER never solves problems live. It names the issue, assigns one owner and a due date, and checks back the next week. Debugging in the room is how a review grows until people stop showing up.
We raised the bar on postmortems as well. High-severity postmortems needed a reviewer at Director or above, and low-severity ones needed Sr. Manager or above. Senior review kept action items from quietly dying.
We also made OER everyone’s meeting: anyone could attend to watch and learn, and facilitation rotated. One OE Guardian from every team attended and carried the follow-ups back to their team. They were usually team leads or senior engineers with a strong bar for operational rigor.
All of this sat inside clear leadership expectations: blameless postmortems, continuous improvement, and performance calibrations that rewarded OE work the same way as complex feature work.
This was a meeting I looked forward to every week. We celebrated the wins, worked through the misses, and kept the steady systems work going that customers felt most.
It paid off. Within 18 months, SLOs in breach fell from 70% to under 4%, even as we rewrote them around customer experience and added many more. Self-inflicted high-sev incidents dropped from 14 a quarter to 3. Pager load fell by more than half, while usage tripled.
Those three rungs made a fourth one possible.
Rung 4 · Sustainability: let it heal
The most important self-healing system we built was the organization itself. I was confident we had built the muscle for leaders and teams to spot problems, fix them early and improve the process without waiting for me.
Once we hit our OE targets, I handed the torch to the leaders I’d hired and coached, who took a few critical endpoints to 5.5 nines for the first time, another 10x reduction in downtime. The testing strategy we set up front let the team ship a multi-quarter query-planner rewrite with near-zero regressions, validated in shadow mode against hundreds of millions of real operations.
This rung also shows you the engineering culture on an ordinary day, when no one is watching. Rewriting the core path was a calculated risk: shadow testing limited the blast radius, and blameless postmortems made it safe to try. We added one field to the postmortem template, “is this a repeat?”, and repeats became rare enough that each one got its own review. And the unglamorous investments kept compounding. The staging environment that engineers told me looked nothing like prod was rebuilt to mirror it in everything but scale. Every change proved itself there before it reached customers, and that kept self-inflicted incidents down long after the project wrapped.
The same ladder, different problems
The same ladder applies to other horizontal problems. At Stripe, we used it for cloud cost efficiency. Solis, our cost-attribution platform, made the spend visible (observability). Hackathons, efficiency metrics and leaderboards gave teams the means and the targets (predictability). Budget reviews with the CTO and Finance made the targets stick: no team could run more than 2% over its monthly budget without exec sign-off (accountability). We bent the curve, and cloud spend grew slower than the business it served.
As more of the SDLC becomes agentic, it applies to our newest actors too. An agent that remediates an incident needs observability to know what broke. It needs predictable runbooks, guardrails and context to act within. And it needs an accountable owner for when it gets things wrong. Self-healing is the top rung, and it’s only safe once the three below it hold. If you only do one thing this quarter, start at the bottom and get a customer-impact signal for every incident.
The rule I’d leave you with: don’t skip a rung. What does accountability look like when the change was written by an agent?
Thanks to Alex Mann for feedback on this post.