Hot Sites

Whoever watches the daily evening newscast on the tele is likely to be confronted not only by the latest proof of the incompetence of politicians, but also by accidents and natural disasters occurring somewhere or other on the planet. Whether it is hurricanes, earth quakes, volcano eruptions, plane crashes or chemistry catastrophes, somewhere in the world something negative is always afoot. This may not overly worry someone who is watching the news while eating his dinner, after all, catastrophes always happen to others. However those who are hit by disaster often face the loss of their livelyhood and normal existence - provided they succeed in saving their lives in the first place. To counter the effect of such accidents and natural disasters large corporations have already for decades been establishing measures such as disaster recovery plans, external computer centres, etc.

Countermeasures available to large corporations, and economically affordable for these, however, are too expensive for small and mid-sized companies. But it becomes problematic and lifethreatening for small and mid-sized companies if this economic predicament becomes an excuse for not doing anything in the line of disaster preparation at all.

lt is the purpose of this article to describe solutions for smaller enterprises, how so-called 'hot sites' can be established and maintained within an affordable budget, as well as which main logistic and administrative factors need to be taken into account when planning and establishing a hol site. We aim to present themes from practical observations and real life cases which we have not seen described in the trade press before.

As indicated above, the evening news on tv supplies enough arguments for an effective disaster planning in companies. This is underpinned by long-time observations by insurance companies. Damage insurance statistics clearly show that not only has the number of catastrophes increased by a considerable factor over the past 20 years, but also the cost of the damage to the individual victims of these catastrophes. These observations are supported by the magnitude of the damage compensation paid out by the insurance companies. In view of these sinister statistics, company executives no longer pose the question of hether an accident will affect their company, but when this will happen, and how much it is going to cost.

The fundamental idea of the hot site concept is to ensure the availability of alternative space with a suitable infrastructure in order to be able to relocate the business or at least fundamental parts of it, to an alternative site as quickly and completely as possible in case of a disaster or catastrophe hitting the normal business site. The idea is to ensure that the disruption of business in a case of emergency is kept as brief as possible. Ideally, customers should not be able to perceive any service interruption at all.

Risk Analysis

Before starting to establish a hol site it is necessary to conduct a Risk Analysis. Because it is impossible to protect yourself against all possible catastrophes (stray comets, invasion by aliens, etc), it is necessary to establish which types of catastrophes are most likely to hit the company. With regard to natural disasters it is naturally necessary to take local factors such as geographic and meteorologic particulars into consideration.

Most of the factors that must be considered in order to establish a hot site can be derived from the risk analysis. One of the most important factors is the required geographical separation between the business site and the hol site. This can vary between a few hundred meters up to several Kilometers. E.g. if the risk analysis indicates that there is a high probability of flooding across a considerable area including the business site then it makes no sense to establish a hot site 50 meters further down the same street. In case of emergency the hot site will be flooded whenever the company site gets submerged.

Geographic factors to consider

Rivers
Problem:
High water formation and distribution
Factors to clarify:
- How fast can a flood situation arise?
- How long before a flood will a warning be issued?
- Which types of local weather situations will generate floods?
- Which types of more global weather situations will lead to flooding?
- How far will the flood extend? Which roads, local and regional, become difficult or impossible to pass?
- What are the highest water levels measured?
Mountains
Problem:
Rock slides, mud slides, avalanches
Factors to clarify:
- The probability of unusual quantities of rain or snow
- Has this type of catastrophe already occurred?
- Which streets and roadways can become unpassable?
Forests
Problem:
Forest fires, falling trees
Factors to clarify:
- The probability of periods of severe draught
- Has this type of event already occurred before?
- Which streets and roadways can become unpassable?
Regional tectonic peculiarities

Earth quakes

The probability of an earth quake in the region
Has this type of event already occurred before?

Regional meteorological peculiarities such as frequent thunderstorms, large amounts of precipitation, extended periods of draught, etc.

This list above lays absolutely no claim to being complete, far from it. lt is only included to give the reader an impression of the types of problem factors needing to be included in a risk analysis.

The risk analysis should be conducted by a professional business continuity practitioner in order to ensure that all probable risks get included in the risk analysis with the correct weighings, and that they are sufficiently well described.

Only after the completion of a risk analysis is it possible to specify the requirements to the hot site.

Organisational analysis

The next step is to conduct an analysis of the business establishing the site. The purpose of this exercise is to establish which business functions must be moved to the hot site in case of emergency, in which sequence the moves should be carried out (women and children first into the life boats), what to bring to the hot site in terms of tools and documents, where the things to bring are situated and last but not least, who is responsible for frictionless coordination and execution of these emergency measures. Furthermore, it is important especially for larger companies to also produce an emergency staffing plan specifying which members of staff will be required to effectuate the move to the hot site, and which members of staff will be required to man the hot site for the duration of the emergency. I am not going to deal with the staffing plan in details here because in small enterprises it is normally easy to gain an overview of the staff and thus possible to handle an emergency situation more flexibly than in a large organisation.

Based on the analysis of the organisation an emergency plan is produced which describes and specifies the fastest way to switch from the regular business site to the hot site. In small enterprises this plan may be only around 30 pages in size, whereas it often reaches around 300 pages for mid-sized companies.

Requirement specification

The specification of requirements is a catalogue outlining all minimum requirements to the hot site in terms of size, layout, infrastructure and logistics. The requirement specification should build on the knowledge gained through the risk analysis and the organisational analysis. The specification will also indirectly determine how much money establishing and maintaining a hot site will cost a small organisation. For this reason it is very important that the emphasis is placed on properties actually required in a situation of emergency and not on unnecessary luxury which will simply unnecessarily increase the expenditure. To express it clearly once again: A hot site must be able to support the execution and maintenance of vital administrative business activities for a limited period of time - it is not supposed to be a second company headquarters.

Size of Premises

Hindsight has often shown the size of the hot site to be of huge significance. The bigger the premises, the more they will allow to be stored and the better the working conditions which it will be possible to establish.

However, the size of the site directly influences the costs of construction and maintenance and must be clearly calculated. Depending on the type and size of the company, it is normally necessary for between four and ten people to be able to work in a hot site in order to keep the company afloat through an emergency. A minimum floor area of 4.5 sq. meter (40 sq. ft.) per person is required. This is sufficient room for a chair, a table and for passageways but does not include space for technical tools such as computers, photo copiers, printers and fax machines. An average of 1 sq. meter (9 sq. ft.) for each piece of technical equipment must be added to the necessary floor area.

Securing the hot site premises

Hot sites suffer from the severe problem that the premises are not used except in emergencies. For this reason it is necessary to secure hot sites more thoroughly than required for normal business sites in order to avoid the site being useless when it actually is needed, in case of emergency. There have been several instances of hot sites being broken into and the tools required for the perpetuation of the business stolen from the premises. This includes computers, telephones, fax machines and telecommunication equipment.

In order to prevent this is it necessary to equip hot sites with secure doors and windows. Installation of effective fencing and a modern alarm system should also be considered.

Door locks is often a particularly tricky theme. Savings are almost always made in this area. You often come across cheap cylinder locks from a building merchant or a DIY store. These are able to withstand a serious attempt to break in for a few seconds at most, because anybody can open them in less than ten seconds with an ordinary and readily available lockpick tool, without leaving any traces or damaging the locks. lt is extremely important to use high quality security cylinders. These are only available from the specialist trade and can easily cost more than GBP100 each.

To prevent nasty surprises, the integrity of the hot site should be verified at irregular intervals, e.g. by a trustworthy person in the vicinity or by a security service provider.

Stored equipment

If equipment that is urgently required to conduct necessary business activities is already stored at the hot site, this will considerably facilitate resumption of business activities during an emergency situation, but doing so does of course not come free.

It is wholeheartedly recommended to store a computer that can be used as a server together with all necessary components to enable the construction of a small network, at the hot site. Furthermore, some telephones and a small switchboard should be kept available. The same goes for a modern or an ISDN card.

Technical Infrastructure

A modicum of technical infrastructure must be available and ready in a hot site. This includes a sufficient number of telephone connections and/or a wireless telephone system and a good power supply backed up by a generously dimensioned UPS.

In the context of emergency use of a hot site it is necessary to store many important and often confidential papers on site. For this reason it is highly recommended that a safe with sufficient capacity is installed and available at the hot site.

Liase with your telecom supplier

How much use is one of the very best hot sites with all its technical facilities if in a situation of emergency all telephone calls continue to end up in your unusable main offices? What is the use of a switchboard with lots of telephone lines if they all have their own numbers that nobody knows about?

In order to be able to implement your emergency plans in this respect it is absolutely indispensible to liase efficiently with the telecoms supplier of your choice. Negotiations must lead to a contractual agreement that guarantees that in case of an emergency the phone numbers used by the company are switched or redirected to the lines installed at the hot site within at most 12 - 24 hours. Sufficiently significant penalties should be built into this contract in case of non-fulfillment.

This will probably not sit well with all telecom service suppliers but in view of the current dependence of companies on telecoms facilities it is difficult to imagine that the telcos will be able to refuse this type of liability in the long run. This type of contract has already existed for a long time in the highly sensitive bank and insurance areas, and this precedent should be used by small and mid-sized companies as well. If negotiations with your telecoms supplier fail to bring about the desired result, we are fortunately in an era of telecom liberalisation. lt is always possible to enter into negotiations with a few suppliers in order to obtain the best possible terms and conditions.

With regard to redirection of ISDN traffic by means of the built-in standard call redirection function, this is certainly not intended for use in emergencies. This type of redirection can normally only be activated from a pre-determined terminal or phone set in the company. In an emergency there may not be access to this terminal or it may not be possible to power it up resulting in a situation in which the redirection cannot be carried through. Thus, for reasons of security the rerouting or redirection of telecoms lines must always be carried out (programmed) by the telecoms supplier on one of their branch or backbone switches.

Fax to email

When it is a question of surviving in a emergency almost everything hinges on the ability to redirect communication channels or to access these from different locations to ascertain their status if nothing else. One of the possible solutions to this problem is nowadays offered by the Internet using services such as 'fax-to-email'. This type of service is offered by several large providers, e.g. AOL and JFAX.

In order to use the 'fax-to-email' facility it is necessary to register to receive a personal fax number. Faxes sent to this number are converted to a graphics file and transmitted to a specified email account. Because it is possible to access mail from anywhere in the world, this type of solution offers a huge flexibility in terms of the ability to receive faxed communications.

Internet Remailer Services

Just as with the 'fax-to-email' services free Internet remailer services are available for email on the Internet. This incorporates obtaining an email address from a service provider together with a facility to redirect incoming electronic mail addressed to this address to any other email address which is convenient. Because it is the user, himself, who controls this redirection directly by accessing a particular web page or sending email to a particular address, the facility offers great flexibility in case of emergency.

GSM Networks

GSM networks should not be discounted when it comes to disaster recovery planning. Because of their large capacity and flexibility they are natural participants in emergency communication concepts. Featuring functions such as call redirection, reception and transmission of fax messages, data transmission (e.g. for Internet access), fax storage in mailboxes, etc, they are among the most flexible solutions in disaster situations. lt is interesting in this context that all these functions may be programmed directly from the handset without involving the telco. The functionality and flexibility of the GSM net is something for each and every network facilities supplier to envy. Both some handsets and the corresponding manuals should be readily available at the hot site.

Additional Storage Capacity

When planning a hot site it makes sense also to analyse the need for additional storage space. This space may be needed to quickly relocate and store important or expensive devices or materials, even though these are not essential to the continuation and execution of basic administrative business functions. Allocation of extra storage space may enable the company to save considerable values, e.g. irreplacable raw materials, in case of a catastrophe.

Continuous Use of a Hot Site

Because it is difficult or impossible to show a reasonable return on investment on preventive security measures, alternative permanent use of a hot site often lies quite high in the minds of top management. This type of plans must in general be discouraged. A provisional site intended for sudden short-term usage is turned into a permanent resource which more often than not turns out to be used for various purposes and unavailable for the originally intended application when the catastrophe finally does occur.

Off-site Storage Capacity

As indicated in the paragraphs above, a hot site must contain a certain minimum of technical infrastructure elements. With a bit of planning it is possible for small companies to utilise these investments in the daily running of the company to increase the degree of information security without perceptible extra costs. A positive spinoff from doing this is that it is verified that the technical equipment installed at the hot site acflually will function in case of an emergency because it is used more or less regularly.

The point is to use the hot site equipment regularly as off-site storage capacity to back-up data. In order to achive a reasonable data security it is advisable even for small companies to produce security back-ups of their data regularly. This task is normally solved by means of streaming tape units, and the back-up tapes are more often than not stored on a shelf right next to the server... Only in rare cases do srnall companies possess a mirror-server on which to store the data of the working server. However, using the available telecoms infrastructure in combination with the hot site server it is now possible to establish a mirror-server in the hot site on which all important information can be transferred and stored on a daily or even continous basis using a simple ISDN dial-up conneciton. lf the hot site is very close to the business site it is even possible to work with radio modems, optical means of transmission (laser) or radio-LAN cards.

By introducing these measures the company not only obtains decentrally stored back-ups of its data but is also able to directly continue work on the remote server using the full set of files, without loosing time, in case of emergency . Hot Site Secrets lf a hot site is also used by a company for decentralised storage of information (e.g as mirror server) it is necessary to introduce special security measures to protect the premises. Particularly when it comes to industrial espionage it is much prefered to steal decentrally stored data from badly secured and unsupervised premises without a great deal of risk, rather than procuring the same data and exposing yourself to a great deal of risk by nicking them from a well guarded and secured corporate building.

Several different security measures are recommended to protect stored data files.

The physical security of the hot site must correspond to the threat.
The data transfer to the mirror server as well as the data storage on the mirror server should be protected by strong cryptography.
Only those among the most trusted employees in the company who must know about the existence of the hot site should be informed about it.

This is even more true with regard to the administration of the hot site access keys. Thus it does not follow that everybody who knows of the existence of the hot site, also has unlimited access to it. With regard to the key administration is must be recommended that a key is deposited with a neutral entity, e.g. an attorney or a public notary.

When the Bailiff Rings Twice...

This far in the article hot sites have only been mentioned in connection with disasters and natural catastrophes. There are, however, other circumstances not belonging to these catastrophe categories which may threaten the existence of a company. Among these are comprehensive searches by police, including confiscation of computers and data.

The threat to the existence of a company in these cases are caused by several factors. As shown by the experience of a number of executives in numerous cases of this type of confiscation of equipment, the police officers conducing the practical disconnection, packaging and transport of the equipment most often work without sufficient professional knowledge. Computers are often simply switched off without first closing down server software in an orderly manner. Network connections are broken by simply pulling out the cables, power cables are physically disconnected from units still under power, etc. In many of the cases referred to here, these incorrect procedures caused loss of data which were essential to the continued existence of the companies.

It is here sufficient to refer to the destructive consequences of simply pulling the plug while a computer is busy re-indexing large databases (SQL databases, data warehousing systems, data mining systems), or when a notebook computer is simply switched off during an encryption procedure. In both these cases it is quite probable that the data can no longer be accessed or even reconstructed. Similar data destruction can happen if an operating system such as Novell Netware or even UNIX and its derivatives is simply switched off while it is running instead of being shut down orderly.

Unprofessional handling of impounded equipment during transport or storage by authorities has often been observed in the past. Hard disks are destroyed because of incorrect handling, or confiscated computers are stored in humid and too cold or too hot environments, leading to the loss of valuable information.

In connection with a confiscation it is normal practice that all data media in sight, including back-up tapes, are taken away. The consequence of these actions are that a company, having suffered this type of raid, has neither a functional computer installation nor back-up tapes at its disposal, enabling it to get up and running again within an acceptable time frame. The loss of data is of course far more serious than the loss of confiscated hardware.

A further problem for the company is the comparatively long duration of the case procedure at both the police and prosecuter level with regard to the empounded computer files. A case is known in which a company director received his notebook computer containing important company data back after 18 months (he was not prosecuted). Even when the hardware is returned there is no way to be certain whether data stored in it are still there, or perhaps have been damaged or even destroyed by unprofessional handling.

The conclusions reached based on these types of (numerous) examples may not please certain circles involved in these activities. lt is, nevertheless, recommended that companies store important information as remote mirror images or decentral back-ups, securely and well protected against access by officials of one kind or another. An off-site storage system installed at a hot site may serve this purpose admirably.

In order to be able to realise these various functions it is essential that only a very small and limited circle of trusted employees know about the hot site. Evidently, no documents or indications of the existence of a hot site should be allowed to exist in the company.

Financial Implications

With regard to the financial investments in a hol site only information based on experience can be given here, because the calculation of a project of this type depend on too many factors and local variables to be generalised. For a small company the initial investment in establishing and furnishing a small hol site will typically range between GBP 2k - 6k. The annual maintenance costs can be kept as low as GBP 3k.