Someone, somewhere within IT cyber security giant CrowdStrike is having the worst day at work in their life.
Early on Friday morning, people awoke to find TV news channels silent, trains cancelled and websites crashing in a massive IT outage impacting tens of millions of computers.
It seemed almost any business running Microsoft’s Windows 10 software experienced a sudden, dramatic failure. Computers appeared to automatically shut themselves down and display the so-called “blue screen of death”.
As dawn broke, Sky News broadcast an error message: “We apologise for the interruption to this broadcast.” It was forced to run archive footage for several hours.
While it is not the end of the world or a nuclear apocalypse – and systems are, slowly, coming back online – one cyber security researcher dubbed the catastrophic blackout “CrowdStrike Doomsday”.
CrowdStrike is a US technology giant. The business is hardly a household name – although Formula 1 fans may recognise it as the sponsor of the Mercedes team and British racing drivers Lewis Hamilton and George Russell.
Crowdstrike is valued at over $80bn (£62bn) and is responsible for developing critical cyber security software used by thousands of businesses.
Its programs are deeply embedded into critical IT systems and networks. CrowdStrike develops a so-called “endpoint detection and response” system, helping large networks to detect and shut down hacking attacks – much like a business-wide antivirus.
Founded in 2011 and headquartered in Texas, its technology has proved wildly popular. It reported revenues of $3bn and over 23,000 corporate customers last year. The company has also become well-known in cyber security circles for its work investigating high-profile hacks, including on the Democratic National Committee in 2015.
As a technology that has to respond to constantly evolving cyber threats, CrowdStrike routinely pushes out updates to its customers with new tools and layers of protection.
However, an apparently rogue file in one of its latest patches appears to have caused a massive IT collapse, according to cyber security experts and CrowdStrike engineers.
The outage has hit customers including airlines, train companies, airports, payment systems, supermarkets, the NHS and even the Houses of Parliament cafe.
On social media, IT managers reported dealing with implosions impacting tens of thousands of machines in their businesses.
Troy Hunt, a well-respected cyber security guru behind the website HaveIBeenPwned, said: “I don’t think it’s too early to call it: this will be the largest IT outage in history.”
While it was largely Microsoft’s Windows 10 systems hit by the IT problem, the root cause of the bug has been attributed to CrowdStrike – specifically an apparently dodgy file in its Falcon Sensor tool.
Posting on X, formerly known as Twitter, Mr Hunt said CrowdStrike is a “massive player [in] the security space” whose technology often has so-called “privileged” access to business networks. This means it has broad control to update and modify a customers’ systems, in theory to remove malware.
However, he said: “This also means that if something goes wrong with an update, it’s able to catastrophically nuke your machine.”
Security experts were already directing their anger at CrowdStrike over its handling of the outage. For several hours, there were no public updates or official statements from the company about the issue – other than those buried within its customer service portal behind a log-in screen.
The company did alert customers, saying: “CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor. Symptoms include hosts experiencing a bug check/blue screen error related to the Falcon Server.”
The business told customers they could undo the error with a manual workaround, however as of mid-morning there had been no global fix or automatic update to remove the bug.
In a post on X, Brody Nisbet, a CrowdStrike executive, said the problem related to a “faulty channel file”, but added that a workaround for customers was not yet working for everybody. He said: “It’s a mess.”
Finally, George Kurtz, CrowdStrike’s founder and chief executive, issued a public statement.
“CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts,” he said.
“This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed.
“Our team is fully mobilised to ensure the security and stability of CrowdStrike customers.”
Shares in CrowdStrike plunged more than 14pc in opening trading in New York, wiping billions of dollars off its value.
The disastrous outage is likely to raise questions over the stability of global IT systems, which are increasingly reliant on little-understood a web of IT infrastructure run by a handful of major companies.
“This is a very, very uncomfortable illustration of the fragility of the world’s core internet infrastructure,” said Ciaran Martin, the former head of the National Cyber Security Centre and a professor at Oxford University’s Blavatnik School of Government.
Mr Martin told The Telegraph he expected the cost of the outage to be “very significant and likely run into the billions” of pounds.
Meanwhile, Prof Alan Woodward, a cyber security expert at the University of Surrey, questioned how such a damaging glitch could have been released by the company. “You just can’t imagine something this awful would happen with proper testing,” he said.
Mr Hunt said on X the problem was similar to fears of the so-called “Y2K” bug in 2000. Engineers feared that millions of computers would crash at the turn of the millennium after failing to keep track of the change of date.
Those fears proved false, as companies effectively updated their systems. However, Mr Hunt said: “This is basically what we were all worried about with Y2K, expect it’s actually happened this time.”
