The CrowdStrike outage on July 19, 2024, has been dubbed the “largest IT outage in history,” affecting approximately 8.5 million Windows devices worldwide and causing widespread disruptions across various sectors.
Here’s a detailed breakdown of the incident. We will keep on updating this on the evolving situation.
Latest Update
| Date | Event |
|---|---|
| July 23, 2024 | Many organisations still grappling with aftermath; CrowdStrike corrects logic error in Channel File 291 |
| July 22-23, 2024 | Major airlines report progress in recovery efforts. |
| July 22, 2024 | Microsoft discloses that approximately 8.5 million Windows devices were affected. |
| July 20, 2024 | Security experts suggest the update likely skipped routine quality checks. |
| July 19, 2024 | Major disruptions reported across various sectors including airlines, healthcare, and financial services. |
| July 19, 2024, 09:45 UTC | CrowdStrike CEO George Kurtz confirms deployment of fix. |
| July 19, 2024, 07:15 UTC | Google confirms CrowdStrike update as the cause of the problem. |
| July 19, 2024, 05:27 UTC | CrowdStrike reverts the content update. |
| July 19, 2024, 04:09-05:27 UTC | Windows systems worldwide begin experiencing crashes and blue screens of death |
| July 19, 2024, 04:09 UTC | CrowdStrike releases faulty sensor configuration update. |
What Took Place
The outage was caused by a faulty software update from CrowdStrike, a prominent US cybersecurity firm. Specifically:
- CrowdStrike released a sensor configuration update to Windows systems on July 19, 2024, at 04:09 UTC.
- This update triggered a logic error in Channel File 291, which controls how Falcon evaluates named pipe execution on Windows systems.
- The logic error resulted in system crashes and blue screens of death (BSOD) on affected devices.
- The issue affected customers running Falcon sensors for Windows version 7.11 and above.
Scenarios of Impact
- Devices experiencing BSOD (Blue Screen of Death) with BitLocker enabled and no access to Recovery Keys.
- Devices experiencing BSOD with BitLocker enabled and access to Recovery Keys.
- Devices experiencing BSOD without BitLocker enabled.
- Devices are still online with CrowdStrike installed.
What to Do
Due to the recent CrowdStrike outage, some hosts may experience a bug called the Blue Screen of Death (BSOD). This guideline provides the necessary steps to manually fix critical servers and restore operations while waiting for an official fix from CrowdStrike.
Current Action:
CrowdStrike Engineering has identified a content deployment related to this issue and has reverted those changes. However, if hosts are still crashing and unable to stay online to receive the Channel File Changes, follow these steps to manually fix the issue on critical servers:
Workaround Steps:
- Boot Windows into Safe Mode or the Windows Recovery Environment.
- Navigate to the C:\Windows\System32\drivers\CrowdStrike directory.
- Locate the file matching C-00000291*.sys, and delete it.
- Boot the host normally.
Additional Considerations
1. Economic Impact: Experts estimate the costs of the outage could exceed $1 billion, although the full extent of the damage is still being assessed.
2. Sector-specific Impacts:
- Airlines: Thousands of flights were cancelled or delayed.
- Healthcare: Some hospital systems reported delays in procedures.
- Emergency Services: Brief disruptions in 911 services were reported in certain regions.
- Government Agencies: Temporary impacts on various public services.
3. Cybersecurity Implications:
- The incident highlights the critical importance of rigorous quality control checks for software updates.
- It underscores the need for robust disaster recovery mechanisms across the tech ecosystem.
4. Future Preparedness:
- Organisations should review and strengthen their IT resilience strategies.
- Diversifying cybersecurity solutions may help mitigate risks associated with single-point failures.
5. Ongoing Investigations:
- CrowdStrike is conducting a thorough root cause analysis to prevent similar incidents in the future.
- Regulatory bodies may scrutinise the incident, potentially leading to new guidelines for cybersecurity firms.
This unprecedented outage serves as a stark reminder of the interconnectedness of global IT systems and the far-reaching consequences of even minor software glitches in critical infrastructure.
At Transputec, we are dedicated to helping our clients overcome these challenges by providing strong security solutions and offering assistance when problems arise. Let’s collaborate to ensure the security of your digital assets and the smooth operation of your business.
Contact our team of experts.
