Not the greatest decision ever made. Credit: Computerworld / IDG A very large thunderstorm is headed for this company’s data center, and there’s reason to worry, says a pilot fish working there. The company is the product of several mergers back in the late ’90s, when it was decided that the newest data center would be the new company’s production data center. That particular data center was, of course, built for a much smaller company than the merged entity, so it had to be immediately expanded. Laments fish: “In the intervening years, the quantity of servers and network and disk hardware in the data center increased rapidly, but the UPS and emergency generator didn’t keep up with the growth.” Nonetheless, with that big storm on the way, a high-level manager overrides the advice of the technical support team and decides to switch the data center over to emergency power, rather than risk the possibility that the city power could be disrupted. The emergency generator is started up and the switchover is made on a Friday evening without a hitch. But the generator had never been tested since it was installed, and the diesel fuel it runs on had been sitting in the tank for a long time, so sludge has accumulated in the bottom of the tank. When the sludge reaches the fuel line, the engine sputters to a stop. Shortly after that, the underpowered UPS runs out of juice. And the entire data center shuts down catastrophically. “The important servers and network were back up and running within 24 hours,” reports fish. “But some of the large databases took days to recover due to data corruption. “And by the way, the city power never had an interruption.” Related content opinion McDonald's serves up a master class in how not to explain a system outage When McDonald's in March suffered a global outage preventing it from accepting payments, it issued a lengthy statement about the incident that was vague, misleading and yet still allowed many of the technical details to be figured out. By Evan Schuman Apr 01, 2024 7 mins Mobile Payment Data Center Industry opinion Failed unsubscribes could be a clue your data's out of control One of the oldest and most frustrating rules about email spam is that the unsubscribe link never works — all it does is confirm your email address is active. But what if the unsubscribe failure is caused by something far more problematic? By Evan Schuman Jan 15, 2024 3 mins Technology Industry Data Privacy Legal news Microsoft’s data centers are going nuclear A job posting suggests that Microsoft is planning to explore the use of small nuclear reactors for its major data centers. By Jon Gold Sep 25, 2023 3 mins Green IT Data Center news analysis The EU Data Act is a lot bigger than iCloud The EU is taking a stand against vendor lock-in for data services, including IoT, connected device, and cloud services. By Jonny Evans Jun 29, 2023 5 mins Small and Medium Business Apple Cloud Computing Podcasts Videos Resources Events SUBSCRIBE TO OUR NEWSLETTER From our editors straight to your inbox Get started by entering your email address below. Please enter a valid email address Subscribe