Thursday, March 13, 2025
Latest:
dhcp server liuxLinux serverNETWORK ADMINISTRATIONS

Thunderbolt Networking is FAST & CHEAP!

Alice AUSTIN

In this video I show how I created a thunderbolt 4 network ring to cluster my Proxmox nodes. The MS-01 is a perfect option for this approach. Is it the ultimate “budget” cluster?

Thanks to Scyto: https://gist.github.com/scyto/76e94832927a89d977ea989da157e9dc

Minis Forum MS-01: https://amzn.to/3V9DkAa
Cable Matters Thunderbolt: https://amzn.to/4bOOZtU
Samsung 980 Pro: https://amzn.to/4dSaxaS
Corsair Vengeance: https://amzn.to/44OehWF

GitHub:
https://github.com/JamesTurland/JimsGarage

Recommended Hardware: https://github.com/JamesTurland/JimsGarage/blob/main/Homelab-Buyer’s-Guide/Q3-2023.md

Discord: https://discord.gg/qW5vEBekz5
Twitter: https://twitter.com/jimsgarage_
Reddit: https://www.reddit.com/user/Jims-Garage
GitHub: https://github.com/JamesTurland/JimsGarage

00:00 – Introduction to Video
00:30 – Unifi Aggregation Switch Integration
03:22 – Thunderbolt 4 Ring Network
05:04 – Thunderbolt 4 Demo (Speed Test)
08:32 – Thunderbolt 4 Configuration
23:32 – Checking
25:12 – Using the Thunderbolt 4 Ring for Proxmox
27:20 – Create Proxmox Cluster

source

by Jim’s Garage

linux dhcp server

Alice AUSTIN

Alice AUSTIN is studying Cisco Systems Engineering. He has passion with both hardware and software and writes articles and reviews for many IT websites.

You May Also Like

Microsoft Windows Server 2003 (Nokia Version) Fixed Text

Alice AUSTIN

ARRUMEI UM SERVIÇO 🧐🤨

Stop trying to plant Server on Bank!!!

Alice AUSTIN

26 thoughts on “Thunderbolt Networking is FAST & CHEAP!”

  • For max performance you will need to set affinity on the performance cores for iperf3. That will get you to 26Gbits/s across the board.

  • I wonder how Apple Thunderbolt4 cables would perform? If that would cure your retransmission error count? Or if the observed netlink speed would be any faster?

  • change auto to allow-hotplug – a few folks found this fixed a bunch of long term and random issues, this is a manual fix you have to do by hand and doing the wrong thing in the proxmox UI can reset this… in which case change it back

  • Like the sweater, but I’m really missing the pastel cardigans. Video isn’t bad either 😉. Appreciate you.

  • Thunderbolt networking is cheap if you have three MS-01s, which are at least $439 USD barebones or $629 USD for the Core i5 12600H version with 32 GB of RAM and a 1 TB SSD. But if you want the top of the line, pre-configured model from Minisforum, each node will set you back $829 USD, which means that three nodes + Thunderbolt cables, will run you closer to $2500 USD in total.

  • Most of us are struggling with IPV4 Especially after reboot. I ended up taking it out and just using IPV6

    Restarting the fee service usually brings ipv4 back up but rebooting the system kills it again

  • nice work! I have mine set up similarly, though i am using ospf instead of fabric and can route other traffic over these links for fail over purposes. I am also exploring to see if some of the comments on sycto’s gist and the proxmox fourms related to it, about using cpu affinity and assigning the thunderbolt-net to only using P-cores, which some have said is helpful.
    Also, instead of clicking on the ‘shell’ on the left in the proxmox ‘folder tree’ list, you can click on the ‘Shell’ button near the top right of the proxmox gui, this will launch (pop out) the shell in its own window, allowing you to have your multiple sessions side by side.
    again, nice work, and feeling very invested in this series as i have the same hardware and doing the same things 😀

  • This is an incredible bit of kit. I wish these MS01s existed when i built my cluster. I'm pretty sure you'll need more then 3 nodes if you plan to run ceph as 3 is the bare minimum count. You don't want split brain, and 3 nodes won't allow for you to perform maintenance.

  • Hi. Would you do a video on how to configure the SR-IOV for the Miniforums. And how to use it in VM's?

  • When setting Ceph, use erasure-coding and not replication to maximize disk space … may possible that you need to create it manually with shell command as if i remember Proxmox only support replication …

  • Can you provide information on cpu utilization if the Thunderbolts are doing maximum traffic? If I get it right, you are kind of sacrificing a little bit of your cpu as an ethernet controler… This might play a roll for making a decision on which version of the machine to by cause dual a 25g nic goes for 50 bucks on ebay…

  • This may be a dumb question but can I do this using USB 3.1 Gen 2? I don't have USB C ports but at least the USB 3.1 Gen 2 has faster speeds than my single 1gb lan port on each node.

  • Why is there a dollar sign in the center of a heart and "thanks?" Why are there affiliate links in the description?
    No dude. You are violating the spirit of the internet. Get these dollar signs out of my face. This is the INTERNET. What are you doing dude?

  • Thank you for breaking it down. Answered all the questions I didn't even know I had!

  • Thanks for the video, I've been wanting to use USB4 between two Linux cluster nodes for a while and was curious about stability. Thunderbolt-net module supports networking thankfully.

  • Pretty sure your V4 issue is the fact that you’re trying to set a/32 on those if you go back to old school sub netting and remember how Anding works. I believe it’ll pop up that each of those is going to think it’s on its own isolated network.

  • That's some great network speed, but I can't help but see you're just using local LVM storage. So no zfs replication or ceph? Seems like a waste to just use the 25 gigabit network just for migrating VM's.

  • hummm mesh networking… now all i need is enough computers to try this on

Comments are closed.