In the early morning of July 19, a software update to CrowdStrike’s Falcon sensor started to cause one of the most extensive IT outages in history, affecting several industry sectors, including financial services, healthcare, transportation, and others.
According to CrowdStrike, the outage stemmed from “a defect found in a Falcon content update for Windows hosts.” At this point, the software update has not affected Mac and Linux systems.
Given the widespread impact this incident has had on industries around the globe, clean-up and response activities are likely to progress into this week. At this time, immediate response should focus on following CrowdStrike’s guidelines for safely restoring critical systems affected by the Falcon update and monitoring CrowdStrike’s security posture.
UpGuard is committed to helping organizations respond to the CrowdStrike incident safely and ensuring they have the information needed to mitigate its effects across their third and fourth-party ecosystems. In particular, Vendor Risk customers with the Fourth Parties module can gain an understanding of how the CrowdStrike incident impacts their fourth-party ecosystem. Refer to the How UpGuard’s Platform Can Help section for additional insights.
Affected by the CrowdStrike incident? Here’s what you should do right now
If you’ve been affected by the CrowdStrike incident, you should first follow the restoration and workaround instructions CrowdStrike published on its official website. The steps include information on what systems are affected and instruct users on how to navigate the issue based on their system’s status and properties.
Next, you should address how this issue has impacted your third-party vendors. Have they been exposed to the incident and followed the proper restoration steps to recover their systems? It’s important to understand that even if your internal systems have not been affected by the incident, the third-party vendors and service providers you rely on may have been.
At this time, it’s also important to assess whether your vendors are still operating with the proper security controls in place. Some businesses may disable Crowdstrike entirely rather than restore their systems to an early version. This could leave your vendors (and you) vulnerable to cyber-attacks and data security threats.
Depending on the prioritization of this incident, companies that rely on CrowdStrike in their supply chain will be at a higher risk than average over the next few days. We have already seen examples of threat actors identifying and targeting CrowdStrike customers.
Here’s a high-level checklist to ensure you're covering the essentials:
- Apply the latest CrowdStrike fix and patches to all affected systems immediately, and regularly check for official updates.
- Keep a close watch on system and security logs for any unusual activity that could indicate lingering issues or exploitation attempts.
- Verify that all critical data backups are current and accessible. Test restore procedures to ensure that data can be recovered quickly and accurately if needed.
- Maintain a high level of vigilance against phishing attempts by training employees to identify suspicious emails and avoid clicking on unknown links or downloading unverified attachments.
- Strengthen access controls, including implementing multi-factor authentication (MFA), to prevent unauthorized access during the recovery phase.
- Establish clear communication channels to keep all stakeholders, including employees, customers, and partners, informed about the incident and recovery efforts. Provide regular updates on the status of the incident and expected resolution timelines.
- Assess which of your vendors have been impacted, and engage with them to understand their response plans and timelines for remediation. Work together to ensure consistent and effective mitigation strategies across the supply chain.
- Update and review your incident response plans to better handle similar supply chain disruptions, ensuring rapid mitigation strategies are in place and regularly tested.
How UpGuard can help speed up your recovery efforts
Organizations are now facing the urgent task of identifying and mitigating the impact on their vendor ecosystem—a process that can be incredibly time-consuming if done manually. Thankfully, technology such as UpGuard’s Vendor Risk rises to the challenge, offering tools to streamline recovery efforts when every moment counts.
Watch this video for an overview how UpGuard can help you respond to the CrowdStrike incident and secure your vendor ecosystem.
Identify impacted vendors and understand concentration risk
With UpGuard, organizations can effortlessly pinpoint which third and fourth-party vendors may be impacted by the CrowdStrike Falcon outage, delivering immediate insights into their exposure with just a few clicks. Vendor Risk customers can leverage the Fourth Party Products filter on the Vendors page, to easily pinpoint affected vendors, ensuring a clear understanding of potential risks. The Fourth Parties page also offers a detailed view of impacted vendors, enhancing visibility into third and fourth-party exposure.
Related: How CISOs should handle fututure CrowdStrike-like breaches.
Understand your vendor's exposure levels with a Crowdstrike incident questionnaire
For vendors that you classify as critical, and where you are missing information about their level of exposure, UpGuard can help streamline additional information gathering with a new dedicated CrowdStrike Incident Questionnaire, now available in the Questionnaire Library. Additionally, all vendor communication is centralized in one location, facilitating more effective team collaboration and work management, and ensuring comprehensive audit tracking if evidence is required in the future.
Review automated alerts for incident updates and changes
The Incidents and News section also offers a comprehensive view of all potentially affected entities, ensuring you stay informed and proactive. When it’s time to update your business and board on the incident status, UpGuard can generate one-click reports that provide a concise summary of the incident's impact. These reports are invaluable for quickly informing internal and external stakeholders, allowing them to understand the situation and make informed decisions without delay.
By utilizing these features, you can confidently navigate the complexities of third and fourth-party risk management during a crisis.
Get the support you need right now
To support global response efforts to this unprecedented incident, UpGuard is enhancing platform access to ensure every organization has the necessary tools at their disposal.
Organizations can take advantage of free 14-days access to the UpGuard Vendor Risk platform to bolster their response efforts.
Existing UpGuard Vendor Risk customers will receive 30 days of free access to our Fourth Parties concentration risk module, while BreachSight customers are offered 14 days of free access to Vendor Risk, enabling them to identify affected vendors and begin remediation efforts.
For more details about this free offer, or to request access, contact UpGuard Support at support@upguard.com.
Fortifying your supply chain moving forward
For many, the CrowdStrike incident may have materialized without warning. However, the truth is that third-party-related incidents are now more common than ever before, and 29% of all data breaches stem from a third-party attack vector.
Despite this startling statistic and the devastating average cost of a data breach (4.45 million), 54% of businesses admit they do not adequately vet their third-party vendors and service providers before onboarding.
While even the best third-party risk management (TPRM) program wouldn’t have prevented the CrowdStrike incident from happening, it would have allowed an organization to understand which of its vendors was affected quickly and prepared them to pursue mitigation as efficiently as possible.
The most effective TPRM programs include the following components:
- Vendor Risk Assessments
- Vendor Security Questionnaires
- Continuous security monitoring
- Detailed reports and dashboards
Establishing a program with these components will empower your organization to swiftly identify, mitigate, and remediate third-party risks before they damage your organization and improve your response time when unavoidable incidents occur.
Related: CISO strategies post-CrowdStrike to safeguard the balance sheet.
What will the long-term fallout of the CrowdStrike incident be?
As the world learns more about the CrowdStrike incident, expect regulatory agencies to respond with increased scrutiny and more intense compliance regulations.
Third-party risk has been in the spotlight for several years, and regulators worldwide reacted swiftly to previous incidents, like SolarWinds, Knight Capital, and MOVEit. The same response is likely after the CrowdStrike incident.
Moving forward, industry regulators will likely require organizations to develop incident response plans further, including systematic procedures to follow depending upon the criticality of the affected vendor. These requirements will also likely require organizations to further collaborate with their third-party vendors to ensure all sensitive data is protected and operations are restored quickly during such incidents.
As a result, third-party risk management will likely become a larger issue for organizations across industries, especially those most recently affected.
CISO Perspective: What can we learn from such a disruptive event?
Perspective provided by Phil Ross, CISO @ UpGuard
The CrowdStrike incident is not the first technological outage to affect global industries and will not be the last. Third-party risk management is an ever-evolving field, and often, the greatest strides come in the wake of incidents like this one. These events are devastating during their fallout but ultimately bring advanced teachings and strategies to third-party risk management and incident response discourses.
In the context of avoiding and reducing the impact of incidents like the CrowdStrike update outage, it's essential to categorize the areas of impact and adopt strategies to minimize disruption. For end-user compute (EUC) devices, such as laptop fleets and fixed or site-based workstations, organizations should delay patches and updates to operating systems, software agents, and applications until they have been tested on representative devices.
Implementing a rapid testing process for urgent updates, especially for protective security software like CrowdStrike's Falcon agent, is crucial. Additionally, ensure mobile device management and roaming devices are configured for mass recovery routines, even if the device cannot complete a normal OS boot.
"To defend against vulnerabilities in widely-used software, organizations need a clear view of their software supply chain. It's not just about reacting when a vulnerability is found but being prepared with actionable insights to either avoid or mitigate the impact."
- Phil Ross (CISO @ UpGuard)
For physical or virtual machines and devices, such as user workstations and servers, a similar approach should be taken. Delay patches and updates until they have been tested on representative machines, and establish a categorization system to prioritize faster verification of less risky updates. Evaluate the risks and benefits of delaying updates for critical systems, especially if widespread impacts are reported.
Applying the 'cattle not pets' principle—remedying infrastructure-as-code build files and instantiating replacement servers without the 'bad update'—is optimal over applying recovery routines to individual servers. For critical services, consider architecting additional diversity into primary and secondary environments, using different third-party components in failover environments where software risks exist.
While such incidents are challenging, they also drive innovation and improvement, ultimately strengthening our ability to manage and mitigate risks in the future.
