Friday
07:00 Crash is detected.
07:00 Troubleshooting of outage is started.
08:30 Connection between Telia and ADEO data center down.
08:40 Connection between Telia and ADEO data center up.
08:45 SAN fills up and goes into "read only" mode.
08:55 Restore of data from backup and SAN in "read only" mode is established.
18:00 Web servers in the air (websites and shops etc.)
21:00 50% of all mailboxes in the air with reduced speed.

Saturday
01:30 85% of all mailboxes in the air with reduced speed.
07:00 Troubleshooting on mailbox DB3 that will not start.
11:00 Start scan of mailbox DB3.
14:00 Copying mailbox DB3 to backup media start.
15:50 Copy of mailbox DB3 to backup media complete.
16:00 Restore of mailbox DB3 from backup.
16:00 Preparation of primary storage and live migration of virtual servers.
16:30 Primary storage operational again and
16:30 Migration of servers begins.
17:30 Expected Restore of mailbox DB3 from backup complete.
18:30 Mount of mailbox DB3.
19:00 All mailboxes in the air with reduced speed.

Sunday
01:00 All servers are expected to go live with partially reduced speed.

Monday
04:00 Normal operation is expected.

NOTE: No data or mail is expected to be lost during a crash. There are large amounts of data to restore and due to the temporary design of our data center, this unfortunately takes a long time.

In connection with a planned switch from Telia to ADEO data center, our primary storage ran out of free space due to a design error. This resulted in a "read only" mode on our primary storage. Therefore, we had to e.g. restore from backup and move ALL servers to a slower secondary storage. We have ordered another SSD storage to be used in connection with the switch from Telia to ADEO so we avoid a similar situation. The new fast SSD storage will be included in the final design as redundant storage for our primary storage. This will give us higher uptime.

We apologize for the inconvenience this has caused.

If you have any questions or problems getting on the system, you are very welcome to contact Jesper Egedal at 20332666.

Sincerely yours