Flash Notice: UPDATE - Widespread Outages Linked to CrowdStrike Software Update

UPDATE (7/22/2024) -

Most are now aware of the CrowdStrike outage and its global impact on various organizations. Currently, threat actors are attempting to exploit this situation. Over the weekend, CrowdStrike released updated remediation steps for specific environments. Microsoft has also provided remediation solutions and their response to the outage in a blog post. 

crowdstrike

This weekend, CrowdStrike released technical details regarding the outage. On July 19, 2024, CrowdStrike released a sensor configuration update for Windows systems as part of its routine operations. These updates are essential for the Falcon platform's protection mechanisms. However, this particular update caused a logic error that led to system crashes and blue screens (BSOD) on affected systems. 

Customers using Falcon sensor for Windows version 7.11 and above, who were online between 04:09 UTC and 05:27 UTC on July 19, 2024, may have been affected by this issue. The faulty update involved a Channel File named 291, located in the directory C:\Windows\System32\drivers\CrowdStrike\, which controls how Falcon evaluates named pipe execution on Windows systems. The update, which counters newly observed malicious named pipes used in cyberattacks, inadvertently caused an operating system crash due to a logic error. Systems running Linux or macOS were not impacted.  

CrowdStrike has since corrected the issue, ensuring no further changes to Channel File 291 beyond the necessary logic update. Please see CrowdStrike’s advisory for updates on remediation steps.  

microsoft

Microsoft published a blog post estimating that the outage affected 8.5 million Windows devices. The company also noted that this number makes up less than one percent of all Windows machines. Microsoft also detailed how they are helping their customers through the CrowdStrike outage. Those details include:  

• Engaging with CrowdStrike to automate their work on developing a solution. Instructions to remedy the situation on Windows endpoints were posted on the Windows Message Center.   
• Developing a recovery tool which uses a USB drive to boot and repair impacted systems.  
• Deploying hundreds of Microsoft engineers and experts to work directly with customers to restore services.   
• Collaborating with other cloud providers and stakeholders, including Google Cloud Platform (GCP) and Amazon Web Services (AWS), to share awareness on the state of impact we are each seeing across the industry and inform ongoing conversations with CrowdStrike and customers.  
• Quickly posting manual remediation documentation and scripts found here. 
• Keeping customers informed of the latest status on the incident through the Azure Status Dashboard here. 

As previously mentioned, threat actors are attempting to exploit the CrowdStrike outage and have created spoofed CrowdStrike domains, as well as spoofed hostnames. Please remain vigilant and be on the lookout for the following indicators of compromise:  

As always, Avertium is here to support our customers and ensure they are secure during this time. Updates regarding the outage can be found in CrowdStrike’s advisory. If you have any questions about the CrowdStrike outage, please contact your Account Executive or Service Delivery Manager. 

overview

What is Happening? 

Avertium is aware of a global IT outage affecting numerous sectors, including banks, airports, healthcare, and media. So far, we know that this outage has been traced to a software update from cybersecurity company CrowdStrike.  

The disruption began in the early hours of Friday, July 19, when devices running Microsoft Windows started displaying Blue Screens of Death (BSODs). Reports of issues quickly spread from Australia to the UK, India, Germany, the Netherlands, and the US. 

The outage has grounded flights, disrupted TV broadcasts, and impacted healthcare services. Key affected sectors include airlines, with United, Delta, and American Airlines issuing a global ground stop. 

What Caused it?  

CrowdStrike engineers found a defect in a Windows update, which has since been isolated. The issue is not a security incident or cyberattack. CrowdStrike CEO George Kurtz confirmed the defect and stated that a fix has been deployed. The issue does not impact Mac or Linux systems.  

Why Did it Happen?  

From Avertium’s perspective, this situation could be a case of an error in the software development process, where new code wasn't properly tested before being released into production. 

This situation highlights the fact that no software solution is perfect on its own. Having experts oversee and coordinate with technology is essential. Security partnerships, like those Avertium offers, enhance security over time, regardless of the latest tools or trends. 

How Does this Impact Avertium’s Customers and What is Avertium Doing?  

Although, Avertium is NOT a CrowdStrike partner and DOES NOT work directly with their software, our Cyber Fusion Centers are fully operational and continue to protect our customers. Avertium’s priority is to ensure you have the support you need during this time. If you have any questions about the CrowdStrike outage, please contact your Account Executive or Service Delivery Manager.  

What Can I Do to Help Keep My Organization Safe?  

During this period of disruption, there is an increased likelihood that malicious actors may attempt to exploit the situation. They often take advantage of such moments when attention is divided to carry out their activities. Here are a few ways you can help keep your organization safe: 

• Stay Alert - Be extra vigilant for any unusual activity or security alerts from other systems. Immediate attention to anomalies can prevent potential incidents. 

• Monitor All Security Systems - Ensure that all other security tools and systems are fully operational and closely monitored during this period of increased risk. 

• Report Suspicious Activity - Promptly report any suspicious activities or security concerns to our team for immediate assistance and intervention. 

avertium's recommendationS

CrowdStrike Engineering has identified a content deployment related to the outage issue and reverted those changes. Please see the following workaround steps: 

• Boot Windows into Safe Mode or the Windows Recovery Environment 
• Navigate to the C:\Windows\System32\drivers\CrowdStrike directory 
• ⁠Locate the file matching “C-00000291*.sys” and delete it. 
• Boot the host normally. 

Please Note: BitLocker encrypted hosts, may require a recovery key.  

CrowdStrike also has workaround steps for public cloud or a similar environment, including virtual: 

Option 1: 

• ​​​​​​​Detach the operating system disk volume from the impacted virtual server 
• Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes 
• Attach/mount the volume to a new virtual server 
• Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory 
• Locate the file matching “C-00000291*.sys,” and delete it. 
• Detach the volume from the new virtual server 
• Reattach the fixed volume to the impacted virtual server 

Option 2: 

• ​​​​​​​Roll back to a snapshot before 0409 UTC.  

Updates regarding the outage can be found in CrowdStrike’s advisory.  

SUPPORTING DOCUMENTATION