0 Comments

As officials at EDS' Rochelle Park, N.J. data center watched Hurricane Floyd's flood waters rise to 17 inches deep over the base computer floor, they grappled with whether to shut down operations there. For a company that handles 160 million combined ATM and POS transactions a month, the crisis forced EDS to implement an around-the-clock disaster relief effort that has drawn accolades from some of its clients. While anticipating being out of commission for up to a week, EDS managed to restore somewhat normal operations within 78 hours, said Bill Duncan, Marketing Director EDS' Consumer Network Services unit. The processor's crisis recovery performance "was stellar," said Kirk Ergang, senior vice president of systems and operations for Cash Station, a Chicago-based regional EFT network. Duncan described the group of 50 employees who dedicated themselves to resolving the shutdown crisis as "heroic." Although Hurricane Floyd had already been downgraded to a tropical storm by the time it reached New Jersey, its intense rain resulted in the kind of flooding that occurs only every century or so. The shutdown in Rochelle Park impacted between 5,000 and 8,000 of the 16,000 ATMs driven by EDS across the U.S. While full operations resumed by Sept. 20, isolated problems stemming from the disaster continued to affect some clients. Settlement was running at least a day behind as of Sept. 23, according to EDS. Floyd ain't pretty Duncan gave this account of the series of events surrounding the crisis: On Sept. 16, Floyd poured up to 11 inches of rain on New Jersey, with the storm peaking at 8 p.m. Approximately 4 feet of water surrounded the Rochelle Park data center. The water that rose above the base computer floor always remained below the raised 36-inch floor that held the actual equipment. But, the threat of electrocution prevailed. At some point, the EDS crew utilized backup generators. However, by 11:30 p.m. officials decided to shut down operations. About one-third of the transactions were "gracefully" brought down. However, the integrity of the systems was retained. Some transactions were switched to EDS' other data centers in Lombard, Ill., Bellevue, Wash. and headquarters in Plano, Texas. A control center, complete with emergency communications, was established at EDS' business center in Morris Plains, N.J. In a forced evacuation by the National Guard, about 30 EDS employees exited the building from a second story window, down a ladder and into an military amphibious vehicle. Another 20 employees remained until they were required to leave at 4 a.m. Sept. 17. However, the remaining 20 were required to leave at 8 a.m. Red Cross escorted them away in boats. Although they weren't allowed inside the center, EDS staff kept an eye on the building from nearby locations to prevent authorities from doing damage inside if another emergency, such as a fire, were to occur. Relief came an hour or two after the water receded later that afternoon. EDS employees were allowed to re-enter the facility by 4 p.m. Staff members determined immediately that the building could be salvaged. After a disaster recovery crew was brought in, cleanup began with mopping and vacuuming. A large dehumidifier that resembled a platform truck with a "gigantic" fan blew moisture out of the building. At 5:30 a.m. Sept. 20, EDS successfully processed its first transaction from the Rochelle Park Center. "We were, I would say, back to normal, probably Monday evening," Duncan said. You've got (e-)mail LeRoy Collins Jr., president and CEO of Armed Forces Financial Network (AFFN), one of EDS' largest clients, said he was in nearly constant communication with EDS headquarters. He, in turn, utilized e-mail extensively to communicate to AFFN's 300-member financial institutions and credit unions scattered throughout the world. "I guess because of this, we had very few incoming phone calls (from members). That was a godsend," Collins said. "It was just a wonderful way to communicate." Collins pointed out that unlike telephone systems, e-mail provides an audit trail, which proved beneficial. E-mail also allowed AFFN officials to review each official communication before it was issued to avoid misstatements. Collins received phone updates from an EDS official every three hours. "She called me through the weekend. Sometimes we talked at my home. Sometimes she was at her home. We had a good flow of information going there," Collins said. Communication was key, Collins said, because AFFN has close to 35,000 participating ATMs in 35 countries. The number of AFFN ATMS impacted by the crisis is unknown. Collins said that he first received a call on Sept. 16 from Navy Federal Credit Union, which had detected a problem. AFFN's network was mostly restored as of Sept. 22, but with some settlement pending. Several other networks were affected, either directly or indirectly, by the crisis. Although not directly impacted, Houston-based PULSE EFT Association's customers experienced some disruption in service, said Warren Coles, executive vice president. PULSE serves 44,000 ATMs and over 2,050 members, "We established certain services with EDS within 24 hours," Coles said. The crisis affected about 10 percent of Cash Station's 6,800 ATMs, said Kate Coleman, senior vice president of planning, marketing and sales. She praised EDS' around the-clock efforts to restore service. Cash Station's Ergang believes EDS was proactive in its crisis recovery actions. "They did an admirable job," he said. EDS clients can continue to receive updates at 10 a.m. and 5 p.m. daily by calling the EDS Client Hotline at 1-800-426-6574.