DNS Server linuxLinux serverNETWORK ADMINISTRATIONS

How Roblox Went Down For 73 Hours

A look into what happened behind the scenes during the longest outage in Roblox history.

Sources:
https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/
https://www.hashicorp.com/resources/how-we-used-the-hashistack-to-transform-the-world-of-roblox
https://roblox.fandom.com/wiki/2021_Roblox_outage/
https://roblox.fandom.com/wiki/Timeline_of_Roblox_history/2016#August_2016
https://news.ycombinator.com/item?id=30013919
https://raft.github.io/
http://www.lmdb.tech/media/20130329-devox-MDB.pdf
http://www.lmdb.tech/doc/
https://db.cs.cmu.edu/mmap-cidr2022/

Chapters:
0:00 Intro
0:33 HashiStack Explanation
4:47 Outage Investigation
8:20 Root Causes Found
11:30 Return to Service
12:19 Slow Leaders
15:56 Resolution

Corrections:
– At 9:44, the default unbuffered channel in Go does not hold any items and has a buffer size of 0. Sends to such a channel are blocked until another goroutine is ready to receive the value. The illustration in the video shows a **buffered channel of size 1** however the overall point still stands.

Music Credits:
– Firecracker by LEMMiNO (https://www.youtube.com/watch?v=ulfoU2MziOc)
– Impact Prelude by Kevin MacLeod
– We’re Finally Landing by Home

source

by Kevin Fang

linux dns server

46 thoughts on “How Roblox Went Down For 73 Hours

  • i swear the community was blaming chipotle [and sometimes slenders bcz it was popular]

  • and for 73 hours, children were their most safest from predators

  • this was really interesting actually

  • Roblox toddlers truly cannot survive even for 3 days, like go get a life or play some other game! I cant even believe i played roblox a few years ago when i was 12, disgusting im shamed. Like literally no one person with many friends or people who dont hate school and socialize play roblox. Glad i stepped out into the fresh grass, life feels new

  • wait roblox was down? i literlly didnt notice XD, it didnt affect me.

  • Thanks! I submitted this in as a suggestion a while ago, never thought it’d be published.

  • My small brain is having trouble comprehending this, so ima just blame the burrito

  • you had to have been there
    the whole roblox community was going utterly mental for 3 days

  • How one new feature dev just made for fun caused 900m$ lost lmao

  • I can imagine that 95% of people who click on this video have no idea what you're talking about

  • These server incident always feels like a surgery where you have to save a person in its current form ASAP

  • I remember that it almost felt like a month. Every hour was an insanity spiral with everyone posting weird theories and explanations. Best one was there was a big spider in the roblox server room 💀

  • So they used a kubernetes like system and it didn’t work. Moral is: Just use k3s

  • Haven't finished the video yet, but this already makes me feel better about the half-day internet outage I fixed at work

  • Imagine how it must feel, starting a free project just as a hobby, and planning to abandon it eventually, then pretty much half the internet starts using it as an important building block to support the web. Now you're just sitting there, and have a choice to make. Stop maintaining the software, and pretty much break half the internet or keep going, getting zero thanks, and zero dollars for your work.

  • hobbyist coder: "this is just a project for fun, don't use it for anyth-"

    giant corporations: "ITS FREE REAL ESTATE"

  • I didn't know this was a summoning salt video towards the end 😂

  • I understand none of this but this is so interesting

  • Finally! These kids need a break for their brains to touch the grass.

  • i think the outage was this + bloxfruits masive update

    Edit: there was over 1m players on bloxfruits, which is insainly high for roblox games

  • I remember when rhis happened i was tweaking from no deepwoken

  • 2000 employees stepping on each other feet? 48 hr+ downtime does not look good on portfolio

  • How does Roblox lose money? They're in the business of farming children, what are they doing wrong?

    The kids make the all content, and then the kids pay for the content using their parents' money. I get hosting all of it has got to be intense but is that really where all the money is lost?

  • I got a place banned for hacking AND PLACES ARENT PLAYERS SO THEY CANT HAC- DID ROBLOX CAUSE THIS
    ROBLOX SHUT DOWN AND THATS WHY??? ROBLOX THAT WAS A FUKIN FALSE BAN FOR A DAY I WILL LEAK YOUR IP SROBOX

  • You know who is a Veteran when they know what happened to Roblox in 2024.
    giv me 100 likes and Imma give you what I mean.

  • Lots of guesses and tries, but shouldn't they generate a flamegraph to see what consul was busy with at the beginning?

Comments are closed.