Fireworks on the Fourth of July are not unknown to Americans, but the Irish office of a US-based publicity firm got a nasty shock on that date this year when the building it occupied went up in smoke.
Fire destroyed the top floor of Huguenot House in Dublin’s St Stephens Green, prompting widespread press coverage as the building is situated next door to the famous Shelbourne Hotel. Much of the coverage focussed on the inconvenience caused to the hotel’s wealthy residents who were evacuated as a precaution, but even though there were no injuries or fatalities, the well being of the companies occupying Huguenot House’s offices was threatened.
One such company was the Irish branch of Edelman Worldwide, a public relations consultancy with offices around the world. The Irish office employs nearly 40 people, the vast majority of whom are typical ‘knowledge workers’, that is, the main tradable commodity of their company is the information, knowledge and contacts that these employees retain, both in their heads and on their computer systems.
It follows that any loss of such information, whether through losing employees or losing computerised records and data would seriously threaten the company’s ability to function properly.
Sure enough, Edelman had a plan prepared for dealing with just such an eventuality, but it had only been drawn up in the last few years.
‘We were lucky that the fire happened when it did and not two years ago,’ said Hugh Gillanders, country manager for Edelman Worldwide’s Irish office. ‘Back then we had no contingency plans at all, and we didn’t even have an IT manager locally. If the fire had occurred then, we would have been in a much worse state.’
Water damage
The position in which Edelman found itself on the evening of ‘04/07’, as the events are now referred to in the internal argot, was bad enough. Fire broke out on the penthouse (sixth) floor of Huguenot House at about 7pm. The few people in the building at that time were quickly evacuated and the Fire Brigade was called.
Although Edelman’s offices were split between the third and fifth floors of the building and were not affected by the fire itself, they suffered severe damage as a result of the firefighters’ necessary action to extinguish the blaze.
‘We were allowed back into the building at about midnight,’ said Paul Lloyd, Edelman’s IT manager, ‘and the scene in our office was like something out of the film Backdraft. Everything was soaked. Only the emergency lights were on and we were being hit by ruined tiles falling on us from the walls and ceiling.’
The desks where their 40 colleagues had worked were soaked. Paper work had largely been destroyed and the desktop PCs were covered in water. However, all Lloyd and Gillanders wanted to get their hands on was the company’s main server, a rack-mounted Compaq Proliant, and the communications cabinet, situated side by side together in the office. Everything else was to be left.
‘We didn’t know the extent of the damage on the client PCs,’ said Lloyd, ‘but all our important corporate data and applications were stored centrally on the server anyway. We threw a coat over the server and the comms cabinet and I took them home. I booted the server up and it worked, so I knew we had escaped the worst.’
As Lloyd also took a daily backup of all the data held on the server and kept the tapes at a location outside the office, it was clear that as long as the principle server was working properly, restoring any lost data should not have caused a major problem.
The following day, a Friday, Edelman’s business continuity plan swung into action. It had been drawn up in collaboration with the ICT departments in the company’s other European offices, but in as much as it applied to the local company’s ICT infrastructure, matters were left firmly in the hands of Paul Lloyd.
First priority was to find alternative accommodation and for the first day, Edelman’s staff were accommodated in a local hotel room, courtesy of the property company from which it rents Huguenot house. All phone lines were temporarily forwarded to employees’ mobile phones and essential work could continue as near to normally as possible.
Another stroke of luck affecting Edelman, according to Gillanders, was that the fire struck just before a weekend. This gave the employees a few vital days to find and occupy suitable new accommodation.
By the Saturday morning, the company had found a new office less than half a mile away from their original premises on which it has taken out a year’s lease. Gillanders expects that it will be at least a year before Huguenot House will be fit for reoccupation. The main ICT infrastructure, namely the server and the communications cabinet were moved in and the company started reassembling its other ICT assets.
As all desktop computers were written off, Edelman had to secure some new equipment quickly. Although some employees had taken their portable computers home on the night of the fire, there was still a need to replenish some notebook as well as desktop machines. Edelman managed to hire 12 notebook and eight desktop machines from specialist IT rental agency Higher Intelligence and these were delivered on the Saturday morning. Over the weekend, the new offices were ‘plumbed in’ with the necessary networks and telecoms wiring. By Monday morning, two working days after the fire, all employees were moved into the new offices.
The majority of Edelman’s work is carried out using standard office productivity applications such as word processing, e-mail and presentation software, although the accounts department has its own specialist financial applications.
Headaches
Most of these applications, with all their relevant data, were available immediately from the rescued server. However, restoring the telecommunications and Internet connectivity proved to be a particular headache.
Edelman’s Dublin office had hitherto been linked to its office in London by a 2Mbit/s leased line on which it ran a virtual private network (VPN). It took 30 days for the carrier eircom to change around the terminations so that the leased line ran to the new offices. This is a standard lead time for such services, but it meant that for the first month, Edelman was having to deal with a greatly curtailed Internet service.
‘Our new office had a satellite connection to our ISPs’ local point of presence,’ said Lloyd, ‘so we used that for our Internet connectivity.’ Alas, this only provided a bandwidth of 256kbit/s, or one eighth as much as Edelman had been used to.
Another consequence of losing the leased line temporarily was that Edelman had to reconfigure its LAN to take account of the satellite link. This meant for example, reconfiguring the server to be a primary domain controller, whereas previously the domain was centred in the London office.
Also, the firewall which had been present in the comms room couldn’t be used without the VPN, so the company had to switch temporarily to using a Web-based e-mail package instead of Outlook which it had used hitherto.
Although the Web-based e-mail allowed staff to continue to communicate as before, it did raise a lot of issues. Facilities like long-standing address books and filters could not be used and Lloyd conceded: ‘Using the Web-based e-mail package instead of Outlook was a big hassle.’ However, once the leased line was reinstalled, normal e-mail service was resumed.
Looking back on how smoothly the transition from gutted building to brand new premises began, both Gillanders and Lloyd acknowledge the importance of having a dedicated IT manager implementing a predetermined plan.
‘The plan had been put in place since shortly after the September 11thtragedy in the US,’ said Lloyd, ‘but it certainly helped that I had been with the company for two years and knew the network backwards. When it came to reconfiguring the network to take account of things like satellite access, I just don’t know how an outsourced service would have been able to cope, no matter how up to date the documentation was.’
Gillanders echoed Lloyd’s sentiment on the benefits of having a local IT manager in charge of a disaster recovery plan. ‘Our quick recovery was the biggest advertisement I can think of for having a dedicated IT manager in house,’ he said.
Lessons
Despite the fact that the recovery plan worked well and that the amount of office downtime was minimal, Edelman has learned lessons from its experience that will feed back into plans for coping with a repeat occurrence. Both Lloyd and Gillanders are candid about their lucky breaks, such as the fact that the crisis effectively took place over a weekend and that the server equipment was not destroyed.
One consequence is that the company will reduce further the risks of the vital server and communications equipment being damaged. ‘In any new premises we will have, the server and communications rack will be housed in a fireproof and waterproof safe,’ said Lloyd. ‘I would also hope that there would be more readily available access to an alternative leased line in the future but that is out of our hands.’
The practice of keeping backup tapes off site has been vindicated, although they weren’t needed this time, and will continue.
Doubtless, Edelman would not care to celebrate another July 4thin quite the same manner as the one just passed, but should the occasion arise, the company will be better prepared than it was last time.
Subscribers 0
Fans 0
Followers 0
Followers