How Roblox Went Down For 73 Hours
A look into what happened behind the scenes during the longest outage in Roblox history.
Sources:
https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/
https://www.hashicorp.com/resources/how-we-used-the-hashistack-to-transform-the-world-of-roblox
https://roblox.fandom.com/wiki/2021_Roblox_outage/
https://roblox.fandom.com/wiki/Timeline_of_Roblox_history/2016#August_2016
https://news.ycombinator.com/item?id=30013919
https://raft.github.io/
http://www.lmdb.tech/media/20130329-devox-MDB.pdf
http://www.lmdb.tech/doc/
https://db.cs.cmu.edu/mmap-cidr2022/
Chapters:
0:00 Intro
0:33 HashiStack Explanation
4:47 Outage Investigation
8:20 Root Causes Found
11:30 Return to Service
12:19 Slow Leaders
15:56 Resolution
Corrections:
– At 9:44, the default unbuffered channel in Go does not hold any items and has a buffer size of 0. Sends to such a channel are blocked until another goroutine is ready to receive the value. The illustration in the video shows a **buffered channel of size 1** however the overall point still stands.
Music Credits:
– Firecracker by LEMMiNO (https://www.youtube.com/watch?v=ulfoU2MziOc)
– Impact Prelude by Kevin MacLeod
– We’re Finally Landing by Home
by Kevin Fang
linux dns server
the downtime was crazy
i swear the community was blaming chipotle [and sometimes slenders bcz it was popular]
and for 73 hours, children were their most safest from predators
11:11 who heard Windows 7 Startup Sound?
11:09 Windows 7 shutdown
11:09 who heard Windows 7 Shutdown Sound?
Their first mistake was using anything "HashiCorp"
this was really interesting actually
Let's always do alot of good ❤
Roblox toddlers truly cannot survive even for 3 days, like go get a life or play some other game! I cant even believe i played roblox a few years ago when i was 12, disgusting im shamed. Like literally no one person with many friends or people who dont hate school and socialize play roblox. Glad i stepped out into the fresh grass, life feels new
wait roblox was down? i literlly didnt notice XD, it didnt affect me.
Thanks! I submitted this in as a suggestion a while ago, never thought it’d be published.
slack notificaiton gave me ptsd holy shit
My small brain is having trouble comprehending this, so ima just blame the burrito
you had to have been there
the whole roblox community was going utterly mental for 3 days
Thank you for the great information and entertainment video like always😊
How one new feature dev just made for fun caused 900m$ lost lmao
h
I can imagine that 95% of people who click on this video have no idea what you're talking about
These server incident always feels like a surgery where you have to save a person in its current form ASAP
I remember that it almost felt like a month. Every hour was an insanity spiral with everyone posting weird theories and explanations. Best one was there was a big spider in the roblox server room 💀
So they used a kubernetes like system and it didn’t work. Moral is: Just use k3s
Well done as always
Haven't finished the video yet, but this already makes me feel better about the half-day internet outage I fixed at work
Imagine how it must feel, starting a free project just as a hobby, and planning to abandon it eventually, then pretty much half the internet starts using it as an important building block to support the web. Now you're just sitting there, and have a choice to make. Stop maintaining the software, and pretty much break half the internet or keep going, getting zero thanks, and zero dollars for your work.
hobbyist coder: "this is just a project for fun, don't use it for anyth-"
giant corporations: "ITS FREE REAL ESTATE"
Cuz it's Garbo
YO YOU DID A ROBLOX VID WWWWWW
I didn't know this was a summoning salt video towards the end 😂
I understand none of this but this is so interesting
Finally! These kids need a break for their brains to touch the grass.
i think the outage was this + bloxfruits masive update
Edit: there was over 1m players on bloxfruits, which is insainly high for roblox games
I remember when rhis happened i was tweaking from no deepwoken
Holy fuck this is insane i love this
2000 employees stepping on each other feet? 48 hr+ downtime does not look good on portfolio
11:28 sadly
How does Roblox lose money? They're in the business of farming children, what are they doing wrong?
The kids make the all content, and then the kids pay for the content using their parents' money. I get hosting all of it has got to be intense but is that really where all the money is lost?
早上好中国,我现在有Bing Chilling
I got a place banned for hacking AND PLACES ARENT PLAYERS SO THEY CANT HAC- DID ROBLOX CAUSE THIS
ROBLOX SHUT DOWN AND THATS WHY??? ROBLOX THAT WAS A FUKIN FALSE BAN FOR A DAY I WILL LEAK YOUR IP SROBOX
nothing like using a hobby project for a massive corporate application :3
this is so cool
wake up honey! Kevin Fang just dropped a new video.
You know who is a Veteran when they know what happened to Roblox in 2024.
giv me 100 likes and Imma give you what I mean.
fireship?
Lots of guesses and tries, but shouldn't they generate a flamegraph to see what consul was busy with at the beginning?
Yeah i remember that crash