How do you measure and improve the quality of an engineering organisation?
60-second answer
I use a balanced set of metrics: delivery metrics like DORA, which are deployment frequency, lead time, change failure rate and time to restore; quality outcomes like escaped defects by severity, customer-reported issues and incident counts; engineering health like test suite duration, flaky rate, coverage of critical journeys and code review turnaround; and user-facing signals like availability, performance SLOs and support tickets. I track trends, set targets per team, and run improvement initiatives with clear owners, then show results to leadership in business terms such as fewer rollbacks and faster releases.
Explanation
Improvement loop:
- Baseline the metrics.
- Pick the two biggest pain points (e.g., change failure rate, suite duration).
- Run focused initiatives (contract testing rollout, flaky-test program).
- Measure after 1–2 quarters; share wins and lessons.
Beware Goodhart's law: metrics become targets and get gamed. Use pairs that balance each other (speed vs stability).
Example
A platform team cut change failure rate from 18% to 6% in two quarters by introducing PR-level API tests and canary releases, while deployment frequency doubled.
Sample code
SELECT date_trunc('month', deployed_at) AS month,
COUNT(*) AS deployments,
COUNT(*) FILTER (WHERE caused_incident OR rolled_back) AS failed,
ROUND(100.0 * COUNT(*) FILTER (WHERE caused_incident OR rolled_back) / COUNT(*), 1) AS cfr_pct
FROM deployments
GROUP BY 1 ORDER BY 1;