Proxmox on Erying Motherboards! – Testing Hybrid CPUs in Virtual Machines

40 thoughts on “Proxmox on Erying Motherboards! – Testing Hybrid CPUs in Virtual Machines”

• Love this channel. I mean who doesn't like craft beer and pc hardware? 😀

• hey Jeff, im running a Unraid server. and im using a i5-13400 and I have no issues what so ever with vms or with docker and vms assign to either p cores or e core. if none where set it will do I for me.

• @Craft Computing Can you try with split_lock_detect=off in grub? Split_lock_detect is a kernel "feature" intended to prevent DoS attacks from less optimized programs that aren't aware of cache coherence.

• 1) You are on the internet.
  2) You said something.
  this does of course means that:
  3) You are WRONG!!!!!

• Can't wait for my new order of 2 glasses and coffee tumbler.

• 160 dollars, where?i read 260

• I'll never understand why some people go rabid leaping to the defense of a multinational billion-dollar company like Intel or VMWare. They don't really need your help, they're doing pretty well.

• We use HyperV(mostly), VMware and a few others. And at least for us we did not manage to get HyperV to actually use eCores on either Win10 or Win11. seems to be a problem with the images we are provided with (and they already have many other problems like being rather bloated to over 40GB without the actual source or any dev-tools).
  And for us that is kind of a problem cause for whatever reason the higherups decided that software development is done on local VMs on laptops ….. where they limited us to low-power Intel systesm. Yeah i got an 15 1250U … 2p8e … most of the performance can not be used -.-

• anyway, you were right, h8rs gonna h8.

• I've been using Proxmox for a couple of months. I love it. I even bought a Ryzen 7 based Nuc style PC. It's a decent spec little machine with 32 gig of RAM and a 1TB nvme SSD. I bunged an old SATA SSD in for snapshots/backups and general extra storage.

  I don't have any particular need for powerful VMs. I only need one decent gaming PC, and I have one, so don't need gaming VM's. I also don't currently do much with home automation, so don't need a home automation server.

  So, what do I use Proxmox for? Simple. Experiments. I tend to learn best when I mess around. VMs are ideal for that, as I can spin one up based on what I need at the time. If I need two or three, I can spin up two or three. I can also destroy them if I need the space back..

  I've got various Linux VMs, an as I want to teach myself Active Directory maintenance, including imaging, I've also got a Windows Server VM configured as a PDC and running "Theopenem". This is a decent imaging system for Windows and Linux that offers a basic ability to manage the imaged machines. Much like a very cut down System Center. It is also free.

  Proxmox is ideal for this. I'm not too worried about backups because there isn't really anything on the server I need to keep. Long term, I think I need to look at storing the backups somewhere else, but I will do that as and when needed.

• It might be a little misleading that you connects i9 with ECC and chipsets B760 and Z790 because this combo do not support ECC 🙂 So I guess to have ECC supported you should have there w680 chipset which is "HAHAHAHHAAHAHA" when we look at the pricing of motherboards with this chipset 🙂

• One thing about E-Cores, there are stupidly Expensive when you run Microsoft Server. You pay per Core not per Thread for Microsoft. Packages in 2's and minimum 8 Packages.
  And worse for all: it counts the Hypervisor NOT the VM. And small businesses run mostly Micrsoft Servers as it is easier to manage. So a MS Licence for a Threadripper 7955WX costs less then a i7-14700k Licence with 20/28. About 300 USD per double Licence. SO if you are running two on Premise Nodes with the 7955WX and have a common MS Setup with 2 Hypervisors, 2 DC, 1App Server 1 Dataserver all in 2022 you pay around 1200 USD extra. When you run MS Server 2019 it goes even up in pricing as you could run Hypervisor 2019 but need to pay for 32 Cores. So 4000 USD Extra. Going back to the E-2488 Processors, they are stupid. 8 P – Cores and no E-Cores 700USD … AMD starts with 7945wx and 12 p-Cores 1200USD. If intel had a E-2400 series with 12-P cores it would be a great Prozessor or a E-2400 with 8 P-Cores and 8 E-Cores like the OLD i7-13700k had, which was literally one of the best small business Server on Intel side, when running an MS Enviroment. With a Sweet maximum of 192GB, instead the 128GB of Memory for AMD. The i7-14700k is stupid and the E-2488 is stupid. Intel has nice Produkts in the D or W Series. And yes my Company is in the Market for a new Small MS Server atm. As i said the Hypervisor is irrelavant of Promox, Hyper-V, VM-Ware. Intresting is the Guest System and Cores of the Metal.

• I have heard that the amount of data that a video card passes through to and from the motherboard is not as much as people think. So, having a a degraded slot for the video card (like the one that has the sticker next to it) may only have a minor effect on gaming performance. It's worth a try to compare a cpu on that board with a given card, then switch the gpu and cpu to a gaming motherboard and see if there is much improvement.

• So, perusing the comments, the instances where this works all seems to be desktop parts, not laptop parts bolted onto desktop motherboards.
  I'd be very curious if you redid this same test but with a desktop CPU if the same crashing/instability ensued. I suspect it will work just fine.
  And yes, I get that the point is to reduce power consumption, but the combination of parts you're using is anything but standard.

  It may well be that the "great deal" is because it's flaky under the scenarios you want to use it for; I've seen mobile CPUs bolted on to Desktop motherboards for years but there is invariably a catch of one sort or another. RAM, expandability, PCIe lanes, or stability in specific areas. I think it's why the larger motherboard makers don't bother with them and only the big OEM like Dell HP & Lenovo bother with it for their ultra small form factor machines.
  If it turns out that it just needs some attention from the devs, great, but at the moment (time of the video) it's not a usable solution out of the box, while a CPU like those E core-less Xeons while slower, works just fine.

  As an aside, you might be surprised how many smaller shops are using VMWare for their virtualization needs, simply because it's easier to find people trained to setup and manage it and if need be replace them with someone with the same training.
  My firm relies on VMWare and we're right in the 3000-5000 user mark. From what I understand, the team was not at all satisfied with Hyper-V by comparison, but they do revisit periodically to see if it gets any better.

• I'm surprised you don't have a desktop processor with e cores for a cleaner test.

• Wonderful test! have you ever thought to test it on a router or server with Intel Ultra9?

• VMware may have 45% of the market–for now. But Broadcom seems to be doing pretty much everything they possibly can to get that number to go down as fast as possible. So this might be a non-issue by the time you post on the topic again…

• Let's see how many vmware users whining about efficiency cores you get going forward. Pretty soon those same users won't be using vmware. I haven't done exact same tests as you, but I haven't had any hard lock ups on 12600T on an Asus motherboard. I would be really curious if your issues are hardware related. Honestly it could just be the lack of bios adjustments which aren't available with those motherboards.

• I have the same erying Engineering Sample board .before using this board I was curious on how bad that pci 2.0 x8 would actually be for gaming. So I plugged in a Rx 5700xt and somehow my gpu reported pci 4.0×8 even though the sticker says 2.0 . I was able to get smiliar framerates to my Ryzen 5 3600 in gaming

  Also I also encounter weird crashes with that engineering sample under Debian Linux using docket. even though it ran fine on windows . Since proxmox is based on Debian I expect the issue to be the same .

  I’d love to hear if you work out some of these issues. I’m tinkering on it Aswell. Have you ever looked into the hidden bios options? (Toggle by ctrl+f8) maybe there could be a fix but I haven’t found something yet .

  I will rather buy a new board before I’ll ever touch windows server so I’m really desperate to find a fix

• Are the Intel E-Cores still not capable of floating point? If so, that could explain your problem. You were doing some FP instructions, got swapped to a new core, and blam – SIGILL.

• Well, at least I feel less guilty spending a lot of money in older and less efficient systems. All tho I'm having a lot of trouble running VMs for gaming, or running VMs at all that for who's is interested to the idea I'm just gonna say to run Windows Natively on multiple machines and MAYBE try to emulate an other inside of windows. I changed an incredible amount if motherboards, I had to swap my 4x2tb ssd in zfe raid for better ones that I'm using without redundancy or the system would hang, and I still have problems with my 10900KF and 4060 that stutter A LOT inside games. I've bought a Gigabyte z590 based motherboard to try again with an 11900KF and rtx 4060 for my main gaming vm plus an rtx a2000 for my second gaming/photo editing machine. If everything goes smoothly I might try to add the third GPU for a third VM dedicated to AI tasks. My Zimaboard is doing much better with proxmox with 2 containers and 1 VM running everything Important I need, but I still am very unsure about a good backup plan to put in place if my single ssd fails, and with network related services I already feel the pain.

• How do cheap-charlie governments steal processor cycles for their super-computing nodes.? Australian seti-at-home didn't need proxmox and arm-android didn't need extra pins… i-3,5,7,9, type -see

• Did you ever think that your instability problems might not be Proxmox, but board and cpu problems?? The second cpu seem to be more stable….could a better board that isn't so "weird" be an idea to work with??

  You could also be working with a flaw in the mobile CPU architecture that may not be a problem with desktop chips.

  If you reran these tests with a more mainstream motherboard and desktop chips, you make it different results….

  Thank you for the video;
  Monte

• Yeah intel should of done the opposite for these small server CPUs. I'd much rather have 16 E cores than just 8 P cores. Better yet just make them with the same core counts as their desktop cpus but all E cores, having 24 E cores in a single cpu would be pretty dope for a small server.

• That motherboard + laptop cpu seems like a very good deal, only thing is that it supports PCIE 2.0, which is a major issue.

• In my country pretty much every company over 20 emplpyees uses VMware.

• How about running all containers vs VM ? and TrueNas as the front end ?

• Got a great deal on an HP mini PC with i7-12700T.
  Works great for ProxMox, no issues whatsoever.

• Best year for the Lions since i was born! I miss Barry Sanders but I thought we really had a chance 😞

• You might want to reconsider testing with a legit motherboard, not one of those chinese mash ups? Other than that thank you for the video!

• I use Proxmox on NUC 13 i7-1360P (4P/8E/16TH) without any issue. But I didn't try to fully load system with benchmarks.

• Would that brew be good for the showerbeer subreddit?? Asking for a friend lol

• Did I miss something on the Erying Boards? I have an older engineering sample i5 board from them, which I also got for around 160 US$, but lately they cost around 600-800 US$. Jeff said, he got them for 160 US$. What are the 12th and 13th gen Erying boards costs for you?

• Interesting! My 13600K system (on Gigabyte Z690 Aero D) has been rock solid running Proxmox (multiple weeks uptime without issue). After watching this I decided to give it a little test. With one VM and some containers running in the background (though not doing much), I decided to spin up two VMs each with 10 CPU cores (maxing out the 20 threads of the 13600K with 6 P cores with HT and 8 E cores without HT) and I ran Cinebench r15 multicore on both at the same time (multiple runs) and the runs completed without any issues. CPU usage goes right up close to 100% on the system, but didn't seem to freeze or crash. May be something with the Erying boards (or the mobile CPUs they're using) that's causing an issue.

• Could it be that the SSD you have is faulty? Also I assume you are running everything stock. Xmp also not good to have on in this case or at least not during initial testing

• I have been running Proxmox on Intel NUC12 with i5-1240P for a month with no issues. At this moment I running 9 VMs without CPU Affinity, but only assign 1-4 vCores to each VM

• Sorry Jeff, your outfit didn't age well.😃

• I see erying board doing erying things. I've used two of their boards and am batting 0-2 on reliability and compatability.

• Another possibility is that the dies being used for the xeon, having the GPU and/or E cores disabled, are dies that failed GPU and/or E core testing. and have little use in the consumer market..

• I have been using Hyper-V with i9-13900H (Minisforum MS-01) and I encounter no stability issue at all. Yes, the performance load balancing is odd and sometimes heavy workload stuck at E cores, but at least I have nothing to worry about if I just want to put more VM running. Normal operation in Win10 inside Hyper-V guest is also very snappy, much snappier than my previous E-2244 / i7-8700.

Comments are closed.