A misconfigured content update released by CrowdStrike late on Thursday inadvertently triggered worldwide outages across Microsoft Windows systems, taking many of the world’s most essential services offline. CrowdStrike was attempting to update content that their Falcon Sensor uses to perform real-time threat detection and endpoint protection by monitoring system activities that identify suspicious behavior to prevent cyber attacks. The content update contains logic designed to fine-tune the detection of malicious activities and is based on the latest threat intelligence CrowdStrike collects on a real-time, continuous basis.
Initiate in Safe Mode
Begin by starting any affected machine in safe mode. This step is crucial because the Falcon Sensor software, which needs updating, is embedded within a specific subdirectory of the Windows operating system. Booting into safe mode ensures access to this subdirectory for the necessary updates. Without entering safe mode, the operating system’s regular functions, which may include the malfunctioning Falcon Sensor, could prevent you from accessing and modifying the files you need to update. Initiating in safe mode is often the first step recommended in most Windows troubleshooting scenarios because it loads the minimum required drivers and software necessary to run the operating system.
The outage was first spotted in Australia, with Windows machines crashing and displaying the Blue Screen of Death (BSOD). The faulty update triggered a Windows blackout worldwide, impacting dozens of airports, airlines, banking institutions, and service companies that all rely on Windows-based systems to operate their businesses. Hundreds of thousands of travelers were stranded in airports around the world. Approximately 2,600 U.S. flights had been canceled as of Friday afternoon, and more than 4,200 flights had been canceled globally based on FlightAware data as reported by the Wall Street Journal.
Acquire Recovery Key
If the affected PC employs BitLocker or another full-disk encryption (FDE) software, you will need the recovery key for each machine. This step ensures you can access and modify the encrypted data required for recovery. BitLocker and other FDE tools encrypt the entire drive, preventing unauthorized access to the data even if the physical storage device is tampered with or stolen. Therefore, having the recovery key is crucial; without it, you will be unable to access the necessary subdirectories and files to apply the required updates and fixes.
IT teams are in for a long weekend and a tough July, as many cloud-based configurations will require individualized updates for every customer running a cloud-based system. Give IT teams a break and, if possible, postpone any large-scale projects until the misconfiguration can be solved. Acquiring the recovery keys might involve dealing with administrative processes and securely storing these keys once obtained. These keys are sensitive and should be handled with the highest level of security to prevent any potential misuse.
Access Subdirectory
Once in safe mode, navigate to the subdirectory where the Falcon Sensor software is embedded. This will be within the Windows operating system’s directory structure. This subdirectory holds the files and executables critical for the Falcon Sensor’s operation, and finding it is essential for applying updates designed to fix the misconfiguration. Often the subdirectory paths may be different based on the system configuration, but standard practices involve accessing paths like “C:Program FilesCrowdStrike”.
The effects of the IT outage also spread across the Microsoft Azure cloud platform. Azure customers complained that they were “experiencing unresponsiveness and startup failures on Windows machines using the CrowdStrike Falcon agent, affecting both on-premises and various cloud platforms.” Azure Health Status shows the outage still impacts Azure virtual machines across the four regions of America, Europe, Asia-Pacific, and the Middle East, and Africa. Therefore, accessing the correct subdirectory in safe mode allows for a targeted approach to correcting the influence of the misconfiguration, providing a route to address the problem systematically.
Perform Required Updates
Execute the necessary updates on the Falcon Sensor software within the accessed subdirectory. This involves updating the misconfigured elements to restore normal functionality. These updates are often scripts or patches provided by the software vendor – in this case, CrowdStrike – and they will generally address the specific issues caused by the initial misconfiguration. Deleting or overwriting outdated files and replacing them with the updated versions can often resolve these issues.
Outage needs to be a call to action for greater cyber resilience. The more cyber resilient a business is, the greater the ability to anticipate, withstand, and recover from a wide variety of adverse conditions, including attacks, intrusion, and compromises. It’s often on CISOs to get cyber resilience right as a core part of their roles in senior management and, increasingly, on boards. Following the required updates, ensuring that the changes take effect without conflict is critical. This, too, needs to be tested and verified by accessing various system functions and confirming that the previous issues no longer manifest.
Reboot System
After updates are completed, restart the machine. This will apply the changes and hopefully resolve the issues caused by the misconfigured update. Rebooting is an essential step to ensure the new configurations take effect across the system. The updated software components replace the malfunctioning parts initiated in safe mode, thus enabling the system to return to normal operational status.
This week’s global outage is what a nation-state attack would look like if a nation’s cybersecurity was weak or didn’t exist. To get a glimpse into what’s at stake when it comes to national cyber resilience and cyber defense, check out the recently released 2024 Annual Threat Assessment of the U.S. Intelligence Community. Rebooting the system allows these newly implemented updates to be integrated into the Windows operating environment, ensuring seamless functionality and interaction with other software components.
Validate Operation
Upon restarting, verify that the system operates correctly. Ensure that the Falcon Sensor software is functioning as expected and that the Windows environment is stable. This step usually involves monitoring system performance, running diagnostic tools, and conducting routine checks to ensure no residual issues remain. It’s crucial to verify that the Falcon Sensor software now accurately detects and mitigates threats, as intended.
Kurtz continues to post updates across social media platforms X and LinkedIn. In the most recent X post below, he commits to providing a root cause analysis of how the outage happened. In the world of security, one must always be prepared for the unexpected and have an incident plan for those surprise events. Monitoring tools can offer deeper insights into the machine’s performance and report any anomalies that could indicate remaining issues or new ones that arise from the updates applied.
Repeat for Other Machines
Late on Thursday, a misconfigured content update released by CrowdStrike inadvertently caused widespread outages on Microsoft Windows systems globally, impacting many vital services around the world. CrowdStrike’s attempt to update the content used by their Falcon Sensor for real-time threat detection and endpoint protection led to these disruptions. Falcon Sensor actively monitors system activities to detect suspicious behavior, aiming to thwart cyber attacks. The specific content update involved logic meant to refine the detection of malicious activities, drawing on the latest threat intelligence continuously gathered by CrowdStrike.
The disruption underlines the critical importance of rigorous testing and oversight in deploying security updates, as even a minor error can have far-reaching consequences. CrowdStrike has since been working to rectify the issue and assure its users that such incidents will be mitigated in the future. The situation serves as a reminder of the complexities involved in maintaining cybersecurity, where any misstep can lead to significant operational disruptions, underscoring the need for robust quality control measures.