The Pipeline That Only Fails on Weekends
Non-deterministic pipeline failures are the hardest to fix. Here's why they happen and what to do about them.
You get the alert Sunday morning. The pipeline failed at 2:47 AM. You check the logs — nothing obvious. You re-run it manually and it passes. You close the ticket and go back to sleep.
This happens every few weeks. Nobody has time to chase it down. The pipeline “usually works.”
That’s the problem.
Non-Determinism Is a Design Flaw
Pipelines that fail intermittently aren’t randomly unlucky. They have a structural reason — you just haven’t found it yet. The most common culprits:
Volume-sensitive logic. Your transform runs fine all week when data trickles in. Saturday night, the batch is 4x larger. The temp table fills up. The join times out. The job dies. Nothing in your code explicitly breaks — it just can’t handle what the weekend data looks like.
External dependencies without SLAs. Your pipeline calls an API, reads from a source system, or hits a shared database. During business hours, that system is healthy. At 2 AM Sunday, it’s mid-maintenance, rate-limited, or simply slower. Your pipeline assumed it would always respond in under 5 seconds.
Time-based assumptions. You’re filtering on today, calculating week boundaries, or partitioning by date. Somewhere in that logic, there’s a timezone assumption baked in. UTC vs EST. Month rollover. Daylight saving transition. These surface on Saturdays and Mondays, rarely Tuesday afternoons.
Stale statistics. Your query planner last updated its statistics on a weekday with weekday data distribution. The weekend run looks different enough that the planner picks a bad execution plan. Same query, different performance, intermittent timeout.
Fix the Problem, Not the Symptom
Re-running the job when it fails is not a fix. It’s a habit that keeps you from understanding your system.
When an intermittent failure surfaces:
- Reproduce it. Find the data or conditions that triggered it. Don’t just re-run — understand why it failed that run.
- Add observability. Log input row counts, API response times, partition sizes. Make the failure visible before it becomes an outage.
- Set explicit limits. If your job can’t handle 10M rows safely, make that a hard guard, not an assumption.
- Test on production-shaped data. Weekend data is often bigger, dirtier, or structured differently. Your test suite probably doesn’t know that.
”It Usually Works” Is a Liability
Stakeholders don’t hear “it usually works” as reassurance. They hear “we don’t understand it.” And they’re right.
A pipeline you can’t reason about is a pipeline you can’t maintain, scale, or hand off. Track down the intermittent failure. It’s always something specific.
The Sunday alert is the system telling you something. Listen to it.