Lessons from the CrowdStrike Outage: Moving Forward by Building Sustainable Systems

Author: Melanie Thomas

Last Friday’s CrowdStrike outage was the result of a cybersecurity update gone wrong, resulting in chaos on a global scale with an estimated 8.5 million computers around the world impacted.

As one of the worst tech outages in history, it would be easy to focus on the strife as everyone from airlines to hospitals to banks were impacted. Instead, we need to look at this as a call to consider how to build systems and processes for resilience while not breaking the budget and increasing the workload on your existing team.

Reflecting on the events of the CrowdStrike outage, there are several lessons we can take away to build sustainable systems.

Protecting Against Potential Security Vulnerabilities

The root cause of CrowdStrike’s “faulty update” for Windows-based hosts likely lies somewhere in the development CI/CD pipeline. The existing checks or other QA misses had caused the issue to roll out as part of an incremental update process.

Mistakes happen, and technologies are available to help assess your Software Development Life Cycle (SDLC) from static code analysis, registries, infrastructure as code, applications, and APIs for security vulnerabilities and potential human ones. 

From the initial review of the potentially corrupted file, it’s not likely that this was due to AI-generated and pushed code. (It would not be a technology or security article if we did not mention AI, but to our future overlords’ credit, this one was not on them.)

EDR: Turning Off Automatic Updates

Nearly all Endpoint Detection and Response (EDR) tools have an incremental update type of setting, just under different terminology or control. It’s up to the controlling entity to enable various settings to control this across their hosts. I’ve always recommended the best practice of turning off any automatic update for critical systems; your ‘crown jewels’ should be purposefully controlled.

Slightly less critical systems shouldn’t be automatically updated until a decent sample set has been tested and validated. While this is easier said than done in significantly large enterprises, the proper tooling and change control can make this efficient.

For more minor ‘incremental’ updates, I’d recommend only enabling lower criticality hosts for incremental updates and never allowing this untested update across most of your host base.

We may see endpoint vendors take heed of the example being set, providing additional time and resources to QA and process control for their incremental and full updates before release. Unfortunately, there are issues between endpoint solutions updates and Windows updates, especially for Windows 10 and 11 operating systems, but never to the scale we see now.

"We need to look at this as a call to consider how to build systems and processes for resilience while not breaking the budget and increasing the workload on your existing team."

CrowdStrike Customers: Avoid Any Sudden Moves 

CrowdStrike customers should still be vigilant. While the Crowdstrike outage WAS NOT a malicious intent event, malicious actors will take advantage of the situation. As expected, malicious actors posing as CrowdStrike or other advanced support services have been reported. Others are uploading and spreading malicious files, which is noted to be a hotfix solution, but they’re infecting them with ransomware or malware instead.

Remember that knee-jerk reactions can result in trouble during times of high stress. Trying to default to an ‘easy button’ or unverified hack could result in even more catastrophic results. While not the sexiest response, this is why IT and Security teams should have vetted processes and procedures for such a type of ‘disaster’ recovery response.

Operational, financial, and reputational impacts will still be assessed over potentially weeks or months, but existing CrowdStrike customers may not have a big appetite to replace this tech yet. There are many competitors in this part of the industry. Still, competing sales teams shouldn’t take this event as an opportunity to downplay CrowdStrike or push to have it replaced across impacted organizations. Rather, we should all be positioning and encouraging all organizations to ensure instead, they are getting a handle on compensating controls cyber/IT monitoring systems, and IT remediation, which will be more top of mind.

VDI and DaaS: A Silver Lining

Who affected by the CrowdStrike outage has seen the earliest silver lining in this ordeal? Organizations that have moved to using Virtual Desktop Infrastructures (VDI) and Desktop-as-a-Service (DaaS) are the winners.

This situation reminds us that a smaller reliance on a single set of traditional hosts but centrally controlled with the ability to manage the master image for persistent or non-persistent desktops can offer a degree of protection. Plus, you can free up your team by relying on a third-party team of professionals to manage the underlying infrastructure and services.

CrowdStrike Outage: Where Do We Go from Here?

Moving forward, we may see slower but more purposeful system updates and ‘enhanced’ testing and QA processes across the industry. This could mean more organizations begin paying attention to change control processes, balancing the need to validate even incremental updates with the security or operational impact, and, of course, increased attention to releases’ development and security impact.

We may also see some ambitious legislative attempts emerge, with vendors like CrowdStrike having various implementations across all types of critical controls. With increasing attention to compliance and regulations in the past few years, an event like this will surely help to move some needles.

Finally, when you return to the office or log in this week, show extra love and appreciation for your IT and Security teams, especially if you were impacted by the CrowdStrike outage. These brave individuals are the ones helping keep things up and running that we often can take for granted.

Bridgepointe’s cybersecurity and networking experts can help you assess potential risks to your business continuity. Get started by booking a complimentary cybersecurity audit now. 

TAGS

Share:

Experience the Bridgepointe Way

Start today with a no-obligation consultation with one of our experts.

Table of Contents

Related Blog Posts