Make the most of a unexpected hardware crash.
Our office has three advisors using laptops that all run windows. All of our programs are installed and run from the laptops. All of our data and databases actually reside on a primary server that is networked to the three laptops, our printers and the scanners. It has two separate drives (a RAID array system that creates a duplicate of all our data) and runs Linux. We also have a secondary server with one drive that is our dedicated mail server and it runs Microsoft Exchange. We use Microsoft Outlook for our contact management, mail and calendar functions. Our Outlook files reside on the secondary mail server. The Exchange program allows us to have access to each other's calendars, provides internet access to our email and facilitates email and calendar synchronization with our cell phone/PDA devices.
Our technology consultant maintains a remote backup service for us that has our primary server push an incremental back up twice a day to two different servers in different locations. The backup also pulls our Outlook files from the mail server. Every Sunday night, a full reconciliation is done on both back up servers. We also save a full image of the server's data on the last day of each quarter for a year. We can access the data on these back up servers through a password protected internet site.
This access function allows us to view and download any file from either of the backup servers to any computer with an internet connection. Since one back up runs at noon and the other at midnight, this data is very current. We have tested this system several times by going out to the servers for specific files to verify that our data was being backed up successfully.
Last year we wrote up our disaster recovery plan to address all types of systems failures as well as an inability to access our office space due to events such as a flood or fire. We believed our systems and processes were effective, however, we did not simulate a complete crash in order to test them. We just did not have the guts for that.
Three weeks ago our firm's primary server began to hiccup. It started taking a long time to open and close files and we were getting odd little warnings that didn't make any sense. We notified our consultant but he couldn't find anything wrong.
Then, during the night, one of the drives in our server failed completely. We arrived the next morning to the shrill sound of the server's alarm. Before touching the server, we verified that our remote backups had run as scheduled and viewed the back up files from the night before to ensure that our recent data had been saved. The backup had run at midnight as scheduled, so we were not facing any data loss.
We restarted the server, hoping it would be able to rebuild the faulty drive. No such luck. Not only was the drive in question dead, but the entire server died in front of our eyes trying to reboot. After an appropriate amount of cursing, we decided to try out the disaster recovery procedures from our manual.
The good news is that our secondary mail server was working just fine and although it is older, it has quite a bit of hard drive capacity. Our IT consultant made a complete copy of our last backup from the remote back up server and brought it into the office on a portable hard drive. He then created a new directory on our mail server and copied all of our server data to it. This took several hours because the mail server does not have a USB port so the data had to be copied via a serial port. The only other step required was that we had to manually go into our programs that save to databases on the server and identify the new location of the databases. By the next morning, we were up and running on the secondary server with no loss of data. Our backup program was adjusted to pull from the mail server so we maintained the integrity of our backup data.