For two years, a client's nightly reporting pipeline ran as a single cron job. It worked — until it silently didn't, and nobody noticed for six days.
What went wrong
The job had no retry logic, no alerting on failure, and no idempotency guarantees. A single transient database timeout meant the whole night's reports simply never ran, with no error surfaced anywhere.
What we replaced it with
We moved the pipeline to a proper job queue with per-step retries, dead-letter handling, and Slack alerting on failure. Each step became idempotent, so a retry after a partial failure never double-counted data.
postgresqlperformance