While an incident response is in progress, the CERT team should
be investigating while working toward data and system recovery.
Unfortunately, it is the nature of the breach which dictates the
course of recovery. Having backups or offline, redundant systems
during this time is invaluable.
To recover systems, the response team must bring any downed
systems or applications back online, such as authentication
servers, database servers, and any other production resources.
Having production backup hardware ready for use is highly
recommended, such as extra hard drives, hot-spare servers, and the
like. Ready-made systems should have all production software loaded
and ready for immediate use. Only the most recent and pertinent
data needs to be imported. This ready-made system should be kept
isolated from the rest of the network. If a compromise occurs and
the backup system is a part of the network, then the purpose of
having a backup system is defeated.
System recovery can be a tedious process. In many instances
there are two courses of action from which to choose.
Administrators can perform a clean re-installation of the operating
system on each affected system followed by restoration of all
applications and data. Alternatively, administrators can patch the
offending vulnerabilities and bring the affected system back into
production.
Performing a clean re-installation ensures that the affected
system is cleansed of any trojans, backdoors, or malicious
processes. Re-installation also ensures that any data (if restored
from a trusted backup source) is cleared of any malicious
modifications. The drawback to total system recovery is the time
involved in rebuilding systems from scratch. However, if there is a
hot backup system available for use where
the only action to take is to dump the most recent data, system
downtime is greatly reduced.
Patching affected systems is a more dangerous course of action
and should be undertaken with great caution. The problem with
patching a system instead of reinstalling is determining whether or
not a given system is cleansed of trojans,
security holes, and corrupted data. Most rootkits (programs or packages that a cracker uses
to gain root access to a system), trojan system commands, and shell
environments are designed to hide malicious activities from cursory
audits. If the patch approach is taken, only trusted binaries
should be used (for example, from a mounted, read-only CD-ROM).
