A client's 45-minute data import gradually became a 14-hour ordeal that bled into business hours, tanked dashboard performance, and made the analytics team distrust their own numbers. The fix wasn't what anyone expected.
- Published On
Hello, I'm Luke
Personal blog about my experiences and thoughts on software development.
Latest Posts
- A client's real-time notification service kept crashing every few days. Restarts fixed it temporarily. The root cause was 14,000 zombie WebSocket connections from clients that had disconnected days ago.
- Published On
- A client had fourteen microservices, a 280-line Docker Compose file, and an wiki page last updated eight months ago. Their newest hire spent two days just trying to get the app running on her laptop.
- Published On
- We spent months building CI/CD pipelines, canary deployments, and automated rollbacks. Then someone changed a timeout value in an admin panel and took down the payment service for two hours.
- Published On
- GitHub Actions was out for nearly eleven hours on August 6. On the client project I was embedded in, it took about forty minutes for us to realize we had no Plan B. We couldn't deploy, couldn't run tests, couldn't merge with confidence. A ten-hour lesson in single points of failure.
- Published On