As a quick thought experiment, the next time you are in your data
center, look around, and imagine for a moment that it is gone. And not
just the computers. Imagine that the entire building no longer exists.
Next, imagine that your job is to get as much of the work that was being
done in the data center going in some fashion, some where, as soon as
possible. What would you do?
By thinking about this, you have taken the first step of disaster
recovery. Disaster recovery is the ability to recover from an event
impacting the functioning of your organization's data center as quickly
and completely as possible. The type of disaster may vary, but the end
goal is always the same.
The steps involved in disaster recovery are numerous and
wide-ranging. Here is a high-level overview of the process, along with
key points to keep in mind.
A backup site is vital, but it is still useless without a disaster
recovery plan. A disaster recovery plan dictates every facet of the
disaster recovery process, including but not limited to:
What events denote possible disasters
What people in the organization have the authority to declare
a disaster and thereby put the plan into effect
The sequence of events necessary to prepare the backup site
once a disaster has been declared
The roles and responsibilities of all key personnel with
respect to carrying out the plan
An inventory of the necessary hardware and software required
to restore production
A schedule listing the personnel to staff the backup site,
including a rotation schedule to support ongoing operations
without burning out the disaster team members
The sequence of events necessary to move operations from the
backup site to the restored/new data center
Disaster recovery plans often fill multiple looseleaf binders.
This level of detail is vital because in the event of an emergency,
the plan may well be the only thing left from your previous data
center (other than the last off-site backups, of course) to help you
rebuild and restore operations.
 | Tip |
|---|
| | While disaster recovery plans should be readily available at
your workplace, copies should also be stored off-site. This way, a
disaster that destroys your workplace will not take every copy of
the disaster recovery plan with it. A good place to store a copy is
your off-site backup storage location. If it does not violate your
organization's security policies, copies may also be kept in key
team members' homes, ready for instant use. |
Such an important document deserves serious thought (and possibly
professional assistance to create).
And once such an important document is created, the knowledge it
contains must be tested periodically. Testing a disaster recovery
plan entails going through the actual steps of the plan: going to the
backup site and setting up the temporary data center, running
applications remotely, and resuming normal operations after the
"disaster" is over. Most tests do not attempt to perform 100% of the
tasks in the plan; instead a representative system and application is
selected to be relocated to the backup site, put into production for a
period of time, and returned to normal operation at the end of the
test.
 | Note |
|---|
| | Although it is an overused phrase, a disaster recovery plan must
be a living document; as the data center changes, the plan must be
updated to reflect those changes. In many ways, an out-of-date
disaster recovery plan can be worse than no plan at all, so make it
a point to have regular (quarterly, for example) reviews and updates
of the plan. |
One of the most important aspects of disaster recovery is to have
a location from which the recovery can take place. This location is
known as a backup site. In the event of a
disaster, a backup site is where your data center will be recreated,
and where you will operate from, for the length of the
disaster.
There are three different types of backup sites:
Cold backup sites
Warm backup sites
Hot backup sites
Obviously these terms do not refer to the temperature of the
backup site. Instead, they refer to the effort required to begin
operations at the backup site in the event of a disaster.
A cold backup site is little more than an appropriately configured
space in a building. Everything required to restore service to your
users must be procured and delivered to the site before the process of
recovery can begin. As you can imagine, the delay going from a cold
backup site to full operation can be substantial.
Cold backup sites are the least expensive sites.
A warm backup site is already stocked with hardware representing a
reasonable facsimile of that found in your data center. To restore
service, the last backups from your off-site storage facility must be
delivered, and bare metal restoration completed, before the real work
of recovery can begin.
Hot backup sites have a virtual mirror image of your current data
center, with all systems configured and waiting only for the last
backups of your user data from your off-site storage facility. As you
can imagine, a hot backup site can often be brought up to full
production in no more than a few hours.
A hot backup site is the most expensive approach to disaster
recovery.
Backup sites can come from three different sources:
Companies specializing in providing disaster recovery
services
Other locations owned and operated by your organization
A mutual agreement with another organization to share data
center facilities in the event of a disaster
Each approach has its good and bad points. For example,
contracting with a disaster recovery firm often gives you access to
professionals skilled in guiding organizations through the process of
creating, testing, and implementing a disaster recovery plan. As you
might imagine, these services do not come without cost.
Using space in another facility owned and operated by your
organization can be essentially a zero-cost option, but stocking the
backup site and maintaining its readiness is still an expensive
proposition.
Crafting an agreement to share data centers with another
organization can be extremely inexpensive, but long-term operations
under such conditions are usually not possible, as the host's data
center must still maintain their normal production, making the
situation strained at best.
In the end, the selection of a backup site is a compromise between
cost and your organization's need for the continuation of
production.
Your disaster recovery plan must include methods of procuring the
necessary hardware and software for operations at the backup site. A
professionally-managed backup site may already have everything you
need (or you may need to arrange the procurement and delivery of
specialized materials the site does not have available); on the other
hand, a cold backup site means that a reliable source for every single
item must be identified. Often organizations work with manufacturers
to craft agreements for the speedy delivery of hardware and/or
software in the event of a disaster.
When a disaster is declared, it is necessary to notify your
off-site storage facility for two reasons:
| Tip |
|---|
| | In the event of a disaster, the last backups you have from your
old data center are vitally important. Consider having copies made
before anything else is done, with the originals going back off-site
as soon as possible. |
A data center is not of much use if it is totally disconnected
from the rest of the organization that it serves. Depending on the
disaster recovery plan and the nature of the disaster itself, your
user community might be located miles away from the backup site. In
these cases, good connectivity is vital to restoring
production.
Another kind of connectivity to keep in mind is that of telephone
connectivity. You must ensure that there are sufficient telephone
lines available to handle all verbal communication with your users.
What might have been a simple shout over a cubicle wall may now entail
a long-distance telephone conversation; so plan on more telephone
connectivity than might at first appear necessary.
The problem of staffing a backup site is multi-dimensional. One
aspect of the problem is determining the staffing required to run the
backup data center for as long as necessary. While a skeleton crew
may be able to keep things going for a short period of time, as the
disaster drags on more people will be required to maintain the effort
needed to run under the extraordinary circumstances surrounding a
disaster.
This includes ensuring that personnel have sufficient time off to
unwind and possibly travel back to their homes. If the disaster was
wide-ranging enough to affect peoples' homes and families, additional
time must be allotted to allow them to manage their own disaster
recovery. Temporary lodging near the backup site is necessary, along
with the transportation required to get people to and from the backup
site and their lodgings.
Often a disaster recovery plan includes on-site representative
staff from all parts of the organization's user community. This
depends on the ability of your organization to operate with a remote
data center. If user representatives must work at the backup site,
similar accommodations must be made available for them, as well.
Eventually, all disasters end. The disaster recovery plan must
address this phase as well. The new data center must be outfitted
with all the necessary hardware and software; while this phase often
does not have the time-critical nature of the preparations made when
the disaster was initially declared, backup sites cost money every day
they are in use, so economic concerns dictate that the switchover take
place as quickly as possible.
The last backups from the backup site must be made and delivered
to the new data center. After they are restored onto the new
hardware, production can be switched over to the new data
center.
At this point the backup data center can be decommissioned, with
the disposition of all temporary hardware dictated by the final
section of the plan. Finally, a review of the plan's effectiveness is
held, with any changes recommended by the reviewing committee
integrated into an updated version of the plan.
