Troubleshooting field guide
How to reason from
symptom to mechanism.
Public postmortems often make diagnosis look linear. It rarely is. This guide preserves the working method: define the failure, keep hypotheses provisional, collect discriminating evidence, and change direction when the facts demand it.
How should an IT incident be investigated?
Start with observable symptoms, define the affected and healthy boundaries, form multiple hypotheses, and seek evidence that can disprove them. Record dead ends and turning points. Treat the root cause as the mechanism best supported by the complete evidence—not merely the first fault discovered.
The diagnostic chain
Five stages of evidence-driven diagnosis
- 01
Frame the symptom
Describe what users, services, and telemetry actually reported. Separate observations from explanations.
- 02
Build competing hypotheses
Keep several plausible mechanisms alive long enough to test them. Early confidence is not evidence.
- 03
Find discriminating evidence
Prefer observations that separate one hypothesis from another: healthy boundaries, timing, protocol behavior, dependency state, and controlled comparisons.
- 04
Recognize the turning point
Make the direction change explicit. The decisive moment is often a contradiction that invalidates the original frame.
- 05
Verify the mechanism
Connect the root cause to the observed impact and recovery. Distinguish the trigger, contributing conditions, and restoration work.
Learn from the record
Troubleshooting principles from approved cases
Meta outage analysis
- Breadth of impact does not identify the faulty layer.
- Healthy transport is a meaningful exclusion signal.
Google Gemini outage analysis
- Proving a healthy layer early can cut the problem space sharply.
Coinbase May 7, 2026 outage postmortem
- The initiating event rarely explains the full duration of impact.
- Recovery sequencing is diagnostic data, not just operational housekeeping.
Microsoft Azure OpenAI multi-region latency and failures PIR
- A convincing early theory is still provisional until it survives a second wave of evidence.
- Trigger, amplifier, and visible failure path must be separated.
Microsoft Azure West US 2 power and cooling PIR
- Recovery is a separate troubleshooting problem from failure initiation.
- Physical incidents become software incidents once shared control planes and storage paths are involved.
Cloudflare .de DNSSEC outage response
- Separate preserved behavior from failed behavior early.
- During shared infrastructure incidents, preplanned exception policies accelerate recovery.
Direct answers
Questions about root cause investigation
What is root cause analysis for an IT incident?
Root cause analysis is the evidence-driven process of moving from observed symptoms to the underlying technical or operational mechanism that produced them. A useful analysis preserves competing hypotheses, tests them against evidence, and separates the initiating fault from contributing conditions and recovery constraints.
How do engineers narrow the fault domain during an outage?
Engineers narrow the fault domain by comparing what is failing with what remains healthy, testing boundaries between components, examining high-value telemetry, and updating hypotheses when evidence contradicts the initial explanation.
Why study dead ends in public incident reports?
Dead ends reveal which symptoms were misleading, which assumptions delayed diagnosis, and what evidence finally changed direction. They make the investigation reusable as a learning tool instead of reducing it to a tidy final answer.