...

January 2024 - VPN Outage due to failed certificate renewal

...

• Certificate used for all AnyConnect VPN clients was allowed to expire

  • View file
    name _.aimes.services.crt

    • Common Name: *.aimes.services
      Subject Alternative Names: *.aimes.services, aimes.services
      Organization:
      Organization Unit:
      Locality:
      State:
      Country:
      Valid From: December 19, 2022
      Valid To: January 20, 2024
      Issuer: Go Daddy Secure Certificate Authority - G2, GoDaddy.com, Inc. Write review of GoDaddy
      Key Size: 2048 bit
      Serial Number: dbe0653f36391488

• Noticed around 22:00 Saturday 20th January

• Reported as P1

• Call from support shortly after

  • Support agent said this would be billable out-of-hours support

  • We pointed out this cannot possibly be true given the issue is entirely due to ARO failing to renew a critical certificate. This is basic stuff.

  • Support agent suggested they would look at it “first thing on Monday” and be resolved “by 8:30am”

• Sunday 2st ticket escalated as UKKA management want it sorted before Monday

  • Updated request not acknowledged

• George chased up around 6am Monday

  • No acknowledgement

• 09:24am Monday: “Good morning, the issue is being worked on by Richard Johnston in the network team , we are looking into a potential certificate issue on one of the core Firewalls. Unfortunately I don't have an update on potential fix time at this moment.”

• 10:45am: “We are just in the process of applying a new certificate, the CA has issued this due to some (as yet) unknown issue with the current certificate.”

  • From George: “I don't think it's as-yet unknown, it's that current SSL certificate expired on Saturday at 1:45pm without having been renewed in advance.”

...