By Steve Thoms · Published October 13, 2026 · Confidence: high · Complexity 7/10
Clerk System Outage: PostgreSQL Query Plan Flip Triggered by Auto-ANALYZE Sampling Statistics Error
On February 19, 2026 at 8:11 AM PST, Clerk experienced an approximately 90-minute outage when a routine automatic ANALYZE produced inaccurate NULL-fraction statistics for a column. The column was 99.9996% NULL, but the planner's sample contained only NULLs, so it estimated 100% NULL and built a plan assuming part of the query would return zero non-null rows; it actually returned over 17,000. That frequently-run query then consumed nearly all database resources, request handlers queued, and more than 95% of traffic returned 429s. Responders first blamed a customer's anomalous traffic spike (and blocked it) before tracing the issue to the query plan flip; service recovered when ANALYZE was re-run manually and the planner returned to the prior plan.
Problem statement
On February 19, 2026 at 8:11 AM PST, Clerk experienced an approximately 90-minute outage when a routine automatic ANALYZE produced inaccurate NULL-fraction statistics for a column. The column was 99.9996% NULL, but the planner's sample contained only NULLs, so it estimated 100% NULL and built a plan assuming part of the query would return zero non-null rows; it actually returned over 17,000. That frequently-run query then consumed nearly all database resources, request handlers queued, and more than 95% of traffic returned 429s. Responders first blamed a customer's anomalous traffic spike (and blocked it) before tracing the issue to the query plan flip; service recovered when ANALYZE was re-run manually and the planner returned to the prior plan.
What investigators first believed
- A customer with an anomalous traffic spike was suspected of triggering widespread database issues; manual blocks were added for that customer's traffic (16:35 UTC).
How the diagnosis unfolded
- 01
Escalate and open investigation
16:15 UTC — incident reported and escalated internally; group call opened to identify the root cause.
- 02
Pursue the traffic-spike hypothesis
16:32 UTC — team identified a customer with an anomalous traffic spike coincident with the outage and believed the increased load may have triggered the database issues.
- 03
Intervene on the suspected cause
16:35 UTC — manual blocks added for the customer's traffic; 16:37 UTC — blocks failed to restore the system, so the hypothesis was dropped and investigation continued.
- 04
Re-examine the spike itself
16:50 UTC — the customer was found to have an overly aggressive retry mechanism; the initial failures requiring retry were inside Clerk's infrastructure — the spike was an effect, not the cause.
- 05
Attempt failover
16:55 UTC — automatic Session API failover failed to trigger because the database was technically online, just degraded; alternatives were investigated.
- 06
Enable the new session failover
17:08 UTC — a new failover mechanism for session token generation outside the core Session API was enabled; 17:10 UTC — many users could reach customer apps again, while sign-in and account management APIs stayed inaccessible.
- 07
Isolate the database-level cause and restore the plan
17:25 UTC — root cause determined: an automatic ANALYZE triggered a query plan flip. 17:27 UTC — ANALYZE manually re-executed on the affected table; the query returned to its prior plan and the database began recovering.
- 08
Confirm stability
17:43 UTC — systems stabilizing with the database back to healthy levels; failover disabled. 18:06 UTC — stable performance confirmed across the system; incident moved to monitoring.
What narrowed the fault domain
NULL fraction estimation
Column truly 99.9996% NULL; planner's undersized sample saw only NULLs and estimated 100% NULL, so the plan assumed zero non-null rows where the query actually returned over 17,000.
Frequently-run query amplification
The impacted query runs frequently; its unexpected traversal of 17,000 rows directed nearly all database resources at it, causing request-handler queueing and 429s on more than 95% of traffic.
Degraded-but-online state
Automatic Session API failover did not trigger because the database was degraded rather than down — the failure mode the existing failover was designed for.
Retry-driven traffic spike
The suspect customer's anomalous spike was produced by an aggressive retry mechanism reacting to Clerk's own initial failures.
Blocking the suspected customer's traffic (16:35 UTC) had no effect — the traffic-spike hypothesis was wrong.
Key turning points
- 16:50 UTC — realizing the customer's retry storm was a reaction to Clerk-internal failures shifted suspicion from the customer back to the platform.
- 17:25–17:27 UTC — identifying the ANALYZE-driven query plan flip and manually re-running ANALYZE restored the prior plan and recovered the database.
Root cause
A routine automatic ANALYZE re-estimated a column's NULL fraction from a sample too small to capture its rare non-NULL values. The planner concluded 100% of values were NULL (true value 99.9996%) and built a plan that assumed part of the query would return zero non-null rows; it returned over 17,000. On a frequently-run query, that unexpected traversal exhausted database resources.
Resolution
Manually re-ran ANALYZE at 17:27 UTC, restoring the prior efficient plan; the database recovered and systems stabilized by 17:43 UTC. Later that day the statistics target on the table was increased; that evening the query was refactored so the planner takes a deterministic approach. Clerk is auditing all queries for similar unstable-statistics risk, added dedicated alerting for query plan flips, and reworked Session API failover to trigger on any failure at origin.
Lessons from the response
- Correlation with a customer traffic spike is not causation — the spike was an aggressive retry reaction to the outage, not its cause.
- Failover designed only for full outages misses degraded-but-online states; design failovers that trigger on any origin failure.
- Alert on the failure class directly (query plan flips) rather than on downstream symptoms like 429s.
- Raise the statistics target on high-stakes tables so the planner's samples are large enough to see rare values.
Troubleshooting principles
- 01
Correlation with a traffic spike is not causation.
- 02
Preserve the diagnostic sequence and evidence, not only the final fix.
A plausible-but-wrong culprit (the retry-driven traffic spike) consumed the first half of the incident; the real mechanism requires knowledge of planner statistics and sampling, rewarding evidence over correlation.
Questions answered
What happened in the Clerk System Outage: PostgreSQL Query Plan Flip Triggered by Auto-ANALYZE Sampling Statistics Error incident?
On February 19, 2026 at 8:11 AM PST, Clerk experienced an approximately 90-minute outage when a routine automatic ANALYZE produced inaccurate NULL-fraction statistics for a column. The column was 99.9996% NULL, but the planner's sample contained only NULLs, so it estimated 100% NULL and built a plan assuming part of the query would return zero non-null rows; it actually returned over 17,000. That frequently-run query then consumed nearly all database resources, request handlers queued, and more than 95% of traffic returned 429s. Responders first blamed a customer's anomalous traffic spike (and blocked it) before tracing the issue to the query plan flip; service recovered when ANALYZE was re-run manually and the planner returned to the prior plan.
What was the root cause?
A routine automatic ANALYZE re-estimated a column's NULL fraction from a sample too small to capture its rare non-NULL values. The planner concluded 100% of values were NULL (true value 99.9996%) and built a plan that assumed part of the query would return zero non-null rows; it returned over 17,000. On a frequently-run query, that unexpected traversal exhausted database resources.
How was the root cause discovered?
16:50 UTC — realizing the customer's retry storm was a reaction to Clerk-internal failures shifted suspicion from the customer back to the platform. 17:25–17:27 UTC — identifying the ANALYZE-driven query plan flip and manually re-running ANALYZE restored the prior plan and recovered the database.
What evidence mattered most?
Column truly 99.9996% NULL; planner's undersized sample saw only NULLs and estimated 100% NULL, so the plan assumed zero non-null rows where the query actually returned over 17,000. The impacted query runs frequently; its unexpected traversal of 17,000 rows directed nearly all database resources at it, causing request-handler queueing and 429s on more than 95% of traffic. Automatic Session API failover did not trigger because the database was degraded rather than down — the failure mode the existing failover was designed for.
Which assumptions were wrong?
That increased load from a customer's traffic spike triggered the database issues — disproved at 16:37 UTC when traffic blocks changed nothing.
What delayed recovery?
The approved record describes restoration as follows: Manually re-ran ANALYZE at 17:27 UTC, restoring the prior efficient plan; the database recovered and systems stabilized by 17:43 UTC. Later that day the statistics target on the table was increased; that evening the query was refactored so the planner takes a deterministic approach. Clerk is auditing all queries for similar unstable-statistics risk, added dedicated alerting for query plan flips, and reworked Session API failover to trigger on any failure at origin. It does not separately quantify a recovery delay unless stated in that account.
What should operators learn from this case?
Correlation with a customer traffic spike is not causation — the spike was an aggressive retry reaction to the outage, not its cause. Failover designed only for full outages misses degraded-but-online states; design failovers that trigger on any origin failure. Alert on the failure class directly (query plan flips) rather than on downstream symptoms like 429s.
Apply the diagnostic method
Use the Troubleshooting Field Guide to compare this investigation with the evidence patterns, hypothesis tests, and turning points found across the Casebook.
Open the Troubleshooting Field Guide →Original incident source
Root Cause Casebook separates reported facts from analyst synthesis. This public record was explicitly approved before export.