A New Year's Day power outage in the United States is being blamed for extensive damage done to a data centre operated by the US National Parks Service (NPS).  The centre is located in Denver, Colorado. It houses nearly all of the data the NPS needs to administer America's national parks and monuments. According to sources, the power went out at the data centre at approximately 3pm local time.  The uninterruptible power supplies at the facility kept things going for about 30 minutes, but the local power company was unable to get the power restored until 5pm.  In the interim, the data centre ended up shutting down. The NPS reports that a significant amount of data was corrupted as a result of the shutdown.  When power was eventually restored, an improper restart procedure damaged hardware as well.  No estimate has been provided for when the data centre will be back to normal operations. For the software portion, the NPS immediately contacted Microsoft who went about the process of restoring as much data as they could via off-site backups however Microsoft said significant portions of the system would need to be rebuilt from scratch.  As for the hardware, it is simply a matter of repairing what can be fixed and purchasing replacements for the remainder.

Backup Generators

Because data centre reports demonstrate how costly shutdowns can be, nearly every facility in the world has backup generators operating on diesel fuel.  This enables data centres to remain in full operation for as long as it takes to restore power.  Just as long as a facility has enough fuel on hand, backup generators can provide power endlessly. It seems rather curious that there were no backup generators mentioned in relation to the NPS data centre. One possibility is that there were no generators to speak of which, according to industry standards, is inexcusable.  Another possibility is that generators were on-site but not working properly. In either case, the proper management of a data centre requires every precaution be taken to ensure this sort of thing does not happen.  Power outages are part of the equation that will never be eliminated, thus necessitating things like uninterruptible power supplies, backup generators and other sources of off-the-grid energy. It remains to be seen how long it takes the US National Park Service to get their data centre up and running again.   The rest of us would be wise to learn a lesson from this incident.  No matter how non-critical a data centre might appear, it will instantly become critical should it shut down due to a power outage. In a day and age where backup technologies are extremely affordable, this is something that should never happen.  In light of this knowledge, it will be interesting to see whether the management of the NPS data centre is called into question or not.  It will also be interesting to note whether anyone in a position of policymaking will be subject to disciplinary action.