Crowdstrike analysis: Trivial programming error had global impact
In an extensive and unnecessarily complicated document, the manufacturer explains its errors and names improvement measures for the security software.
Symbolic image of a software bug, the original bug has 21 legs.
(Image: SIMON SHIM / Shutterstock.com)
Three weeks after the major Windows outage caused by CrowdStrike security software, the company has now published its final analysis, explaining all the causes of the disruption. In the twelve-page document, CrowdStrike also explains which countermeasures should prevent such incidents in the future.
"Rapid Response Content", "Sensor Detection Engine", "Content Validator", "Template Type" – Right at the beginning of the "Root Cause Analysis", CrowdStrike throws all kinds of internal company terminology at the reader. At least there is a short summary that explains the error without much preamble.
It is quickly outlined: The faulty update contained 21 data values, one more than the kernel driver expected. It therefore tried to access an invalid memory address – the result is well known.
Videos by heise
What the Fuzz?
The fact that this simple programming error, which could easily be avoided by a runtime check for the correct array size, caused more than eight million blue screens worldwide is surprising and requires a more detailed explanation. CrowdStrike provides this and not only admits to not having tested the fatal update file before the rollout, but also reveals incomplete test routines for other components of its "Falcon" security software.
CrowdStrike stated that the Falcon sensor implemented as a Windows kernel driver was subjected to extensive internal testing, including fuzzing, prior to WHQL (Windows Hardware Quality Labs) certification. The fact that this automatic bombardment of the driver with random data did not reveal a missing array length check is remarkable.
Whether this can be considered "validated, tested and certified", as CrowdStrike CEO George Kurz allegedly claimed during a conference call with investors in March, will now have to be clarified by the courts.
CrowdStrike vows to do better
Kurtz also prefaces the analysis report with a short statement from August 6 in which he expresses his gratitude to loyal customers, apologizes again for the negative effects of the failed update and vows to do better.
This comes in the form of measures that should be taken for granted for software of CrowdStrike's size: There's talk of input checking, array length checking and "a wider range of test criteria". CrowdStrike thus obscures the nature of the error: it is due to carelessness and non-compliance with its own data definitions, not an esoteric, rarely occurring scenario. The Falcon sensor simply could not cope with receiving exactly as much input data as specified in the data definition.
The unchecked update procedure was also much criticized by administrators and security experts. System administrators did not have the option of delaying or preventing updates to the Falcon rule definitions – this should be possible in future. In addition, CrowdStrike will also introduce a gradual update process by means of "canary testing", i.e. testing on a small group of dedicated users and several waves of updates for paying customers.
It is doubtful whether the loyal customers mentioned by CEO Kurtz will continue to include Delta Airlines, which is currently in a public dispute with Microsoft and CrowdStrike.
The CrowdStrike disaster and the lessons learned from it are also the subject of intense discussion in the community of heise security PRO, the specialist service for security professionals. In the community forum, experts are discussing whether a shift from kernel to user space is the solution or whether companies should even do without EDR (Endpoint Detection and Response) software altogether.
(cku)
