Skyscanner : “Kubernetes Fail Compilation: Getting Worse Every Time”

Accident – Death – Obituary News : : 1. “Kubernetes fail compilation gone wrong”
2. “Kubernetes fail compilation disasters escalate”

Learn why Kubernetes engineers are certain that at some point, you will suffer an outage. Despite the sea of uncertainty in the tech industry, outages are inevitable. Discover why failures should be embraced as learning opportunities, essential for maturing and delivering high-quality services reliably. Explore patterns common in Kubernetes cluster failures, such as DNS, networking, and default resource allocation issues. Delve into real incidents from companies like Venafi, JW Player, and Skyscanner, highlighting the importance of post-mortems for learning and improvement. Join Glasskube on their mission to build the next-generation Kubernetes package manager by supporting them on GitHub.

**Major Outage Strikes Reddit on Pi-Day**

Reddit, the popular social media platform, faced a major outage on Pi-Day, March 14th, 2023. The incident lasted a significant 314 minutes and left users across the platform unable to access the site. During this time, visitors were met with error messages, an overwhelmed Snoo mascot, or a blank homepage. This outage came as a shock to Reddit, as they had been focusing on improving availability in recent years.

The root cause of this outage was traced back to an upgrade from Kubernetes version 1.23 to 1.24. This seemingly routine upgrade introduced an unforeseen issue that led to the platform-wide failure. The engineering team at Reddit found themselves in a challenging situation where a rollback, although risky, became the best course of action to resolve the issue.

During the restoration process, additional complications arose due to mismatches in TLS certificates and AWS capacity limits. Despite these challenges, the team successfully navigated these obstacles and reinstated a high-availability control plane for the platform.

Further investigation into the incident revealed that an outdated route reflector configuration for Calico was at the heart of the problem. This configuration, which had become incompatible with Kubernetes 1.24 due to the removal of the “master” node label, led to the platform-wide outage.

In the aftermath of this incident, Reddit acknowledged the importance of improving their pre-production cluster for testing purposes. They also recognized the need for enhanced Kubernetes component lifecycle management tools, the importance of creating more homogeneous environments, and the necessity of increasing their Infrastructure as Code (IaC) and internal technical documentation.

As with many tech companies, Reddit openly shared the details of this outage to help others in the tech community learn from their experience. The tech industry understands that failures and outages are inevitable, and the best defense is to learn from them once they have passed.

**Share Your Outage Experience**

Have you weathered any particularly difficult outages in your tech career? If so, share your experience in the comments below. Your insights and stories can help others navigate similar challenges and learn from your experiences.

**Support Glasskube**

At Glasskube, we are committed to creating valuable content like this and building the next generation package manager for Kubernetes. Your support means a lot to us. If you find value in the work we do, consider giving Glasskube a star on GitHub. Your support helps us continue to create content and tools that benefit the tech community.