Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

https://renalregistry.atlassian.net/wiki/spaces/~150928124/pages/2274394185/Internal+Incident+Post-Mortem

Mid 2023 - Lost server hardware

Mid 2023 - SQL server failure

  • 17/07/2023 10:32 ticket raised

    • “Hi Support,

      Our SQL server (RR-SQL-LIVE - 172.31.158.34) seems to have rebooted itself at 1 am last night and installed an update.

      This update seems to have failed resulting in the error "Cannot recover the master database. SQL Server is unable to run. Restore master from a full backup, repair it, or rebuild it. For more information about how to rebuild the master database, see SQL Server Books Online."

      As a result the SQL service now won't start. Can this please be looked into as a priority?

      We have data backups from 11PM last night (but not the master DB) so reverting the server to yesterday's backup and then restoring the DBs might be an option if there's no quicker way?

      Many Thanks,
      Michael”

  • 17/07/2023 10:51 ticket acknowledged

  • 18/07/2023 15:09 AIMES claim everything is fine due to presence of an unrelated library file

  • 24/07/2023 17:02 call arranged to discuss

  • 27/07/2023 17:49 AIMES raise a ticket with Microsoft

  • 15/08/2023 17:48 ticket escalated

  • 21/08/2023 17:07 AIMES give up, suggest a chargeable rebuild

  • 22/08/2023 16:30 call with Glenn Roberts, AIMES agree to do rebuild for free as they hadn’t realised we first raised the issue so early

  • 18/09/2023 15:55 AIMES chase up as no work done on the rebuild so far

  • 02/10/2023 14:21 works finally begin

  • 17/10/2023 15:58 timeline for data transfer and cutover agreed. Cutover scheduled for 01/11/2023 08:00

...