Severe Degradation Causing Issues with Login and Various Operations

Status - Resolved

Summary - An outage occurred on Aug 1 from 9:55-10:34am MT due to expired TLS certificates associated with deprecated services. Although these services were no longer active, our NGINX ingress controller continued checking them during traffic routing for all Maui services. Despite all currently used services having valid certificates, the presence of the expired ones in the controller’s cache triggered routing issues. We resolved the issue by restarting the ingress controller to clear the stale certificates and this restart and recovery took approximately 25 minutes. All services were then fully restored.

Impact - 

All traffic was refused by the ingress controller from 9:55-10:34am MT causing a disruption in access to Flowhub Maui and several supported applications for most users.

Resolution - 

Restart of the Ingress controller to clear these cached expired certificates was the resolution.

Future Preventions - 

• We’ve created a check for certificate validation in advance of certification expiry. Flowhub has heartbeat monitoring of its services, but those don’t flow through the traffic controller to do their checks.
• We’ve added more FE Latency check alerts to customer support channels