Systemic risk exposed: the Crowdstrike global outage

26 thoughts on “Systemic risk exposed: the Crowdstrike global outage”

• Do you think I should do more videos on security or the intersection of security and AI?

• I have watched many videos about the CloudStrike incident. This is by far my favorite. Dr.Waku your breakdown and analysis is amazing. I particularly appreciate the way you use simple language to convey complex thoughts. Thank you!

• Cease ai. This accident was caused by Ai writing critical code. It will only get worse.

• It's better to fail down, than to fail up. One advantage is 8.5 million machines, could not be hacked!

• Dereferencing NULL Pointers: As you mentioned, this error occurs when a pointer that has not been initialized or has been set to NULL is dereferenced. This leads to system crashes or the notorious Blue Screen of Death (BSOD) because the NULL address (typically address 0) is invalid and not accessible by programs. This kind of error is particularly dangerous and common in kernel mode because of the high privileges and lack of safety nets found in user space.

  Memory Leaks: In kernel mode, memory management errors can be catastrophic. Memory leaks occur when allocated memory is not freed, causing the system to run out of resources. This can lead to performance degradation or system crashes.

  Buffer Overflows: These occur when data exceeds the allocated space, potentially overwriting other important data. This can lead to unexpected behavior, data corruption, security vulnerabilities, or system crashes.

  Race Conditions: These happen when multiple threads or processes operate on shared data without proper synchronization, leading to unpredictable results. In driver code, where operations may be interrupt or device-driven, race conditions can be particularly problematic.

  Invalid Device Handle Usage: Errors can occur when drivers use invalid or previously closed device handles. Such usage can lead to unpredictable behavior or system crashes.

  Improper Synchronization: Failing to correctly synchronize access to shared resources can lead to corrupted data or crashes. This includes improper use of mutexes, semaphores, or other synchronization primitives.

• Is it just a coincidence that a company that named itself "crowdstrike" has positioned itselfed in the business sphere to be able strike at masses of people at once, and then did exactly that? Crowdstrike, did, in fact, strike the crowd. Most people are too brain dead and spiritually decimated to comprehend that these things are done intentionally, both the naming and the action of the company, even when it is right in front of their faces.

• I don't think Michigan First Credit Union is among the world's biggest organizations.

• If it was Deployed during Office Hours in India, then why did it Start affecting Payment Systems in Australia first ?

  In fact the Airports from India to Japan were the last to get affected.

  Now i am curious about the Scheduling of the Deployment.

• So, I can't help myself, as I work with a similar product called Carbon Black EDR, to push back on these channels.
  People were grounded, stranded and had the outage because companies DO NOT support IT. How can I say that, because a tool that is operating at the kernel, you ALWAYS test before you deploy. When you get the produce from the vendor, YOU TEST before deploying to your fleet.

  I initially had some concerns, that maybe an attacker got at crowdstrike's build environment, or that a windows patch was not caught. What it sounds like is the update package was empty, and broke the driver.
  If manually handling this, there would be md5sum sum checks to confirm a bit for bit copying of the files necessary that crowdstrike was sending.

  But here is the thing, stuff breaks. Adding any software is risk. The most secure computer is one that is turned off and at the bottom of the ocean. There is risk in every update and every change.

  So why do these companies trust anyone to upload and configure their CRITICAL systems without testing? Because they don't want to pay for IT staff that will push back and insist on testing.

  Heck, they could have a small segment of machine and sensors that get the auto update and report on failure, before pushing to every other machine.

  But instead, all these Youtube channels are popping up, claiming that it's crowdstrike's fault or windows fault. The fault is not respecting the software, especially the "sharp" software that cyber security products need you be, in order to protect and remediate your systems. Do these same companies cry like this when windows blue screens a segment of the market? No, because more than likely they are testing windows updates. Or worse, they update so late that criminal organizations have been using their systems as a malware distribution system since day 2 of the security update.

  So is it they didn't know what they bought, or they are just completely trusting of everyone who operates in the cloud. None of this is magic. We've been building and breaking computers for years now. We have university programs for god sake.
  So, when an ops, dev, or security engineer makes any sort of noise along the lines of, WE SHOULD TEST ON A SMALL SECTION BEFORE WE FULLY DEPLOYED ACROSS THE ENTERPRISE, don't argue with them and insist on auto updates. Instead, pay them a living wage and have them test before you deploy.

  In the long run, it will always be cheaper for your business. Also you have to endure is someone saying, "Maybe this isn't a good idea for us."

  My $0.02. But also, all the companies that were hit with this, please get you heads out of you ass. There is no way I would trust you with my data if you don't understand a simple test before deploy paradigm.

• Hi @DrWaku, which ivy league institution did you go to?

• Although this was of course a very bad and also carelessly caused outage it still was nowhere near the "global internet outage" that is often titled as. The internet only went down at very few providers, all messenger worked just fine, electricity didn't cut out. Only one of 5 supermarkets in my area had problems (cash only for a few hours) This was not good but certainly no catastrophe.

• This is truly excellent reporting. Outstanding work.
  Thank you.

• When we say that we can understand why the immediate deployment of the update to all systems might make sense, we are pointing to serious defect in the human adaptation to technology. To our armies of brilliant software engineers, now supported by AI capable of writing brilliant code, we need to add armies of human systems engineers.

  When are you consider the unknown, or perhaps even unknowable, state of the future interaction (1) OS code, (2) driver code, (3) hardware built by an uncoordinated collection of manufacturers, (4) data, and (5) encryption (in many cases with lost encryption keys), it is quite obvious that the question of the arrival of update deployment accidents is far more a question of “when” rather than one of “if”.

  This is so obvious in hindsight that our great corporate leaders should be deeply embarrassed by their failure to see this issue and reflect on how an accident my play out in their organizations.

  This is simply not a question of finding the best software solution. At the very least the leaders of our powerful elites should reflect upon the national security vulnerabilities connected with how we have decided to deploy technology. When, for example, you see multiple banks and other great companies keep imploring us to ‘go paperless’, and accept that legal notices have been delivered merely because the deliverer has pressed a Send button at her/his computer, you can understand that the level of dumbness among us human beings is something over which we could easily shed tears.

• 2:41 Oh, crowdstrike was absolutely a cyberattack 💯.
  It's that this cyberattack, unlike Wanacry, that was 13 years in the making.

• Microsoft had to make a 2009 regulatory agreement with EU regulators that required them to allow kernel-level updates by security software providers.

• So Microsoft now is blaming an EU commission decision to address competition concerns that forced them to keep the kernel open to third party vendors, but what i am seeing in the actual document that was published as part of the committee process is that "Microsoft should provide the ability to uninstall internet explorer, Microsoft should allow OEMs to preinstall the browser of their choice without retalation, etc", so this decision Microsoft refers to isnt about granting third parties access to the kernel but it is about letting the end users and OEMs decide which browser they should install on their systems. In general it was about the system not forcing a certain choice of software to their users.

• They should likely be broken up as they are currently a monopoly essentially…

• When I think of Crowdstrike … I cant help but think of the line from "A Few Good Men". "You want me on that wall." (firewall) "You NEED me on that wall." No one will complain as the kernel itself needs protection of this magnitude as the kernel is designed poorly.

• The arsehole politicians in Australia pushing the cashless agenda have become very quiet in the last week.

• This is the most cogent presentation of the issues involved.

• i said it was AGENNNNNNTICCC!!! best get fishing. Mother XXXXX[]

• so dystopian… no worries about the coming agi/asi… lol

• They have kernel access and do not have time for WHQL testing I hear. The kernel thing means that any OS with bad code would crash it might be that the updates for the other OS's were ok.

• When I was a chairlift operator, a guy on my crew quit, by dumping the emergency brake, then taking the handle for the hydraulic brake-release pump (150x) and chucking it into the forest. Was this a disgruntled employee, nobody was listening to about how dangerous CS's code was? You gen Z'ers change jobs all the time, maybe a disgruntled employee's last day…

• I guess you think the Nordstream accident was like the Crowdstrike critical infrastructure accident..

• Amazon AWS RDS didn't want me to do incremental rollouts when I was there 2018-2022 or so. My first couple of rollouts I did do incremental and I actually caught some stuff, but then because of pressures and wanting to follow company alignment, I did all my rollouts single shot.

Comments are closed.