Transcription
I had a whole intro planned. Then a cat decided to steal my chair.
A couple weeks back, I showed off Super Micro's latest microcloud server featuring AMD Epic 405 processors along with Solidime PCI Express 5.0 NVME drives. Today, we're going to put it to work with a Proxmox Seph cluster. Spoiler alert, I might not go back to solo servers ever again.
Welcome back to Craft Computing everyone. As always, I'm Jeff. A quick recap. This is the Super Micro MicroCloud. More specifically, the AS3015MR-H10NR. It's a 3U box with up to 10 of these individual compute nodes, each rocking an AMD epic 4565P 16 core Zen 5 CPU along with 128 GB of DDR5 ECC memory. With a base clock of 4.3 GHz and a boost speed of up to 5.7, this is an insane amount of power in such a small space. Not only that, but having it distributed into up to 10 separate nodes rather than one monolithic server means you get all the redundancy and replication you might need for your organization in one very tidy package.
Today, we're going to set up a Proxmox Seth cluster on the microcloud. This will let us pull some of the resources from each node together, allowing management of every node from a single web UI and allowing for redundancy or even high availability in the services we plan on running. Thanks again to Super Micro for sending over the microcloud for this project. But they weren't the only ones to help out. Solid also sent over the storage for this build in the form of a dozen PS10 7.68 TBTE Gen 5 U.2 NVME drives. While the drives are Gen 5, unfortunately, the microcloud interface is only two lanes instead of four. Even still, a single disc is getting 7500 megabytes per second reads and 6,300 megaby per second writes. and two discs together in a server still means read speeds of over 15,000 megabytes per second.
Unfortunately, I don't get to keep any of this hardware as it's going to be shipped back in a couple of weeks. But thanks again to Super Micro and Solidime for letting me show it off. And while I still have my hands on it, let's build the fastest Proxmox cluster that I've ever put together. I've dabbled with some HomeLab configs here on the channel and built a couple small Proxmox clusters for clients over the last few years, but this is definitely one of the coolest hardware setups I think I've built with so far.
We're going to be using five nodes, mainly because with two 10 GB nicks per node, that's 10 ports and is exactly how many ports that I had free in my 10 GB switch. If I were running this config in production, that is five nodes of AMD Epic 4565P CPUs with a pair of Solid PS10 7.68 TB SSDs, I'd recommend going with at least 25 GB networking to ensure you're getting the most out of your hardware.
Proxmox allows you to configure multiple networks in a cluster to handle different aspects. Networking for your VMs, back-end communication for storage and management, and the like. For today, we're going to be binding both 10 gigabit ports together into a single interface. This can cause some latency issues in production if you need all of your bandwidth for storage transfer between nodes or you have a VM that needs all 10 GB to itself. But for today's demo, a shared 20 GB connection is going to work just fine.
First off, each node needs to have the latest version of Proxmox installed, which at the time of filming is 8.4.5. Well, except for the fact that version 9 launched exactly two days ago from filming, but the instructions are identical between the two versions. As far as I can tell, I'm installing Proxmox onto a single 500 GB SSD in each node, leaving the 7.68 TB drives alone for now. Like I mentioned, each node has a pair of 10 GB network ports, and I've got both connected to a 10 GB switch on a flat VLAN.
Proxmox's network config by default sets up a network bridge and then assigns physical ports to communicate with that. When booting up Proxmox for the first time on these nodes, however, while my static IP address was configured for the bridge, it didn't automatically assign the physical network ports to the bridge interface. I had to manually configure each node to use the 10 GB nicks before I could access the web gooey. So, with a physical keyboard, mouse, and monitor attached to each node, I had to type in IP address show and take note of the physical device names in each node. In my case, I wanted ENP1s 0F0 and 0 F1. Then I opened up etc network and interfaces and then under the VM bridge zero you'll see a line for the bridge ports. Enter your physical network interfaces here and save your changes. Once done, you will need to restart your network configuration. You can either reboot the node entirely or type in system control restart networking.
With the networking all sorted, I will need to update each individual node before we add it to the cluster. Now, since I'm running Proxmox in a home lab, I don't bother paying for a subscription to the Proxmox enterprise update repo. In the web UI on each individual node, click on your server, go to updates, and then repositories, and then click on add. You'll want to add the no subscription repo. Then click on the enterprise repo in the list and click on disable. If running in a production environment, using the enterprise repo is highly recommended as it is a bit more stable and upto-date, but does require a subscription from Proxmox. For home labs or even small and medium business environments, the standard non-subscription route is usually enough. Make sure each server is up to date by running apt update and apt upgrade. And when updates are complete, I recommend rebooting each node. And then we can get started setting up our cluster.
For a cluster setup in Proxmox, you want a minimum of three nodes, especially in our case when we're going to be sharing storage resources across the entire cluster. For a cluster to function, you need what's called a quorum. That is all of the nodes must agree on which node is controlling the cluster. If a single node drops out, the remaining nodes will vote on which node becomes the master. If there are only two nodes remaining, each node will vote for the other, resulting in a tie. And if there is no master, there's no way to tell the nodes where the resources are at, and the whole thing breaks down.
You'll also want to make sure the time and date on your nodes are all perfectly in sync, ideally by running a time server on your local network to avoid any issues. Seph is extraordinarily picky about time and desyncs in time can cause problems such as degraded performance and increased latency, instability, or even nodes and storage becoming completely unavailable.
While you can set up a cluster completely in the web UI, I actually prefer to do it from the terminal. And believe it or not, it's actually a bit easier. The web UI requires copy and pasting secure keys between nodes and can be a bit confusing. In the terminal though, it's literally just a single command on the first node along with a single command and the root password on all the other nodes that you want to join to it. In my setup, I have five nodes named MC1 through MC5. On MC1, I'm going to access the shell and type in PBECM create seph-cluster. This will create a new cluster and add MC1 as a single node. Then I log into MC2 and type in PBECM add and then the IP address of MC1. This process takes about a minute and should result in MC2 being added to the cluster along with both nodes being visible in the web UI. Repeat the PVECM add process on the rest of your nodes and you should see them all appear in the web UI.
Once you've got them all set up, we'll also need to enable SE monitoring on each node. In the web UI, click on each node, go to SE, and then click on install when prompted. This should take less than a minute on each system. Once installed, go to each node, go to seph and then monitor, and click on create. Once complete, each node should be able to monitor cluster health and real-time use, both for compute and for storage.
Speaking of storage, what good is a compute cluster without clustered storage? Like I mentioned at the top of this video, in this cluster, each node is going to house a pair of Solid PS10 Gen 5 NVMe drives. The drives will be configured as one large storage pool, allowing for near instantaneous VM migrations between nodes, along with allowing for both redundancy and resilience in case of either drive or node failure. To set up the drive cluster, we'll have each node report what drives they are making available and then set up our storage pool with those shared drives. Believe it or not, it's actually a very straightforward process that takes just a couple minutes to set up.
A disc in Seph is called an OSD or an object storage damon. To pull drives together, we'll need to define the SSDs in each node as an OSD. Head to each node in the web UI. Go to SE and then OSD and click on create OSD. Select the physical disc you want to make available to Seth from the drop-own menu. Again, in my case, it's going to be the Solid PS10s and then click on create. As you add OSDs, they should start showing up in the Seth OSD tab, listing the node they're hosted on, the OSD type, and the available capacity.
With all of the OSDs made available, it's time to create our Seph storage pool. Head to one of your nodes and go to Seph pools and then click on the create button at the top of the screen. Give your storage pool a name. In my case, I'm going to name it MC store. Similar to ZFS, Seph is a block storage system that distributes data across multiple discs. The size field in the SE setup indicates how many copies of each block of data that you want to keep in the pool, while the minimum size indicates the minimum number of copies of each block that need to exist to keep the pool online. This is essentially the resiliency setup for your SE pool. And the default setting of 3 and two means that for every block of data you write to SE, you will have three individual copies spread across the pool. And the pool needs a minimum of two copies of your data to keep the SE pool online.
There are a number of advantages to running Seph in a cluster like this instead of a dedicated remote storage system, but also a couple downsides. While Seph is extremely resilient, it doesn't do well when it comes to capacity. In the default 32 size configuration, you'll only wind up with 1/3 of your total drive capacity available to you. In my case, I have 77 terabytes worth of NVME drives, but only 24 tab available once the SEP pool has been configured.
In this configuration where Proxmox nodes are directly controlling each Seph OSD, performance is exceptionally good even over their shared 20 Gbit network connection. For our use case here, that is storing VM discs for Proxmox on OSDs hosted by Proxmox. It's an ideal situation as it's natively configured with the Proxmox cluster setup. It makes the SEP pool natively accessible by any node in the cluster and doesn't require any additional hardware to make it all work.
With the SEP pool created, it should automatically be accessible from each of your nodes. And if it does show up, you're basically ready to roll. At least when it comes to creating and running virtual machines in a cluster with a shared SE storage pool. One thing I'd like to note at this stage, and I'm not sure if it's the epic 4565pus or the Gen 5 SE storage, but VM creation is fast, and I mean really fast. Normally on a single Proxmox server, you'll walk through the VM creation wizard and then wait anywhere from 20 to 30 seconds while Proxmox sets everything up in the background with the configuration file and the disk image. On this system, hitting create in the wizard, the VM is ready to turn on basically instantaneously. I've never seen that kind of performance before.
If you're watching this video, I'm sure at this point you probably already know how to install a VM, so I won't bore you with the details here. If you are new though, I do have some more basic Proxmox tutorials that I will link down in the video description. I did go ahead and set up four VMs on each node, mainly because I wanted to show why you might want to run a Proxmox cluster instead of individual servers in some environments and to figure out if this solution is right for you.
Focusing on small and medium business environments, server downtime can mean complete work sophages for your business. If you're running services like Active Directory, inventory management or payment processing databases, ticketing or sales systems, even a few minutes of downtime can add up to a tremendous loss in business or productivity. While most services have options for redundancy, and you should use them, Proxmox also offers high availability modes to automatically transfer and keep alive VMs if a node goes offline. Having both clustered VM hosting and storage all in the same system means less overhead for high availability services and again less hardware than you might have needed in the past.
In a Proxmox cluster with shared and clustered storage, you can live migrate virtual machines with essentially no downtime. Let's say you needed to take a node down for maintenance or upgrades. Simply select the node that you need to shut down, right click on the VM you want to transfer, and click on migrate. Then select the node that you want to move the VM to and click on migrate. Again, since we're on clustered storage, the VM's files technically already exist on the new node. So, the migration process only needs to transfer what is stored in active memory over to the new node. The majority of RAM can be copied while the VM is still active. Once it's all copied, the VM pauses momentarily while the last bit of active memory are copied, as well as checking for any bits that might have changed during the transfer. and the VM spins up on the new node as if nothing ever happened. Live migration makes shuffling of VMs to different nodes pretty much seamless with often less than a second of downtime to move a VM between nodes.
But what about moments of unscheduled maintenance? What happens if you lose a node because of hardware issues, network outage, or power outage? High availability is a feature that allows Proxmox to monitor VMs in a cluster and if it detects that a VM has gone offline, automatically spin it up on another node, again with virtually no downtime. And just to show how well it works, let's set up a VM with high availability. And then I'm going to go pull that node out of the cluster while it's running.
First go to data center in the web guy and then HA or high availability. I've already got VM number 644 on node 5 monitored for high availability. So, let's go ahead and set up number 643. Under resources, click on the add button. This will bring you to a list of all of the VMs in your cluster. I'm going to scroll down and select VM number 643. There are settings for max restart and max relocate, which are the maximum number of tries the cluster will attempt to get the VM running on another node. For right now, we're going to leave these at the default setting of one. Once you're done, click on add. And that's pretty much it. Your VM will now automatically migrate to a healthy node if the cluster detects it's gone offline.
All right, here we are out in the garage by the server rack. And uh you can see the microcloud cluster is sitting right here in the center of it. Now, I did go ahead and configure all four VMs on MC5 as high availability. And this is definitely not the recommended way to test this. I'm literally going to pull MC5 out of the server cluster while it's running. Now, unlike the live migration that I demoed a little bit earlier, this will involve some downtime. The server nodes won't have the ability to transfer the live memory from one server to another. So, they will start up the VM on a new node and have to reboot it. Now, the whole process takes about 2 minutes in its current configuration. mainly because the nodes need to be sure that if a node goes offline, it's actually offline and not just a network hiccup. You don't want another node to spin up the same virtual machine with the same IP address and accessing the same databases or whatever other services you're trying to run as that will create way more problems than it's trying to solve.
But as you can see on the screen right here, we have MC5 and it is currently running. Uh we've got the four virtual machines that we are going to be transferring without any intervention at all to the other nodes as soon as this one goes offline. So without further ado, three, two, one, and node 5 is now offline. So I'm going to go up here to data center. I'm going to click on high availability. And right here is MC5. What's going to happen is it's going to miss its next heartbeat. From that moment, the other nodes will wait 60 seconds. Then they will establish a new master if a new master needs to be established through a quorum and then they will start divvying up the processes or the VMs amongst themselves. So there's the missed heartbeat right there. Uh right now MC2 is the master and a quorum has already been established. So the quorum is not going to change. So after 60 seconds, the nodes will determine amongst themselves which node has the least amount of draw on it right now and it will transfer the VMs in a certain order over there. Now you can also define certain resources that have to be available on a node. For example, if you have a node that requires graphics capability, you can make sure that that only transfers to a node that also has a graphics card on board. So we're about 20 seconds in so far. There's 30 seconds. There's 60. And we should on the next heartbeat start seeing the new VMs pop up. And there they are. Uh looks like MC1 picked up uh two of them. Uh we've got the Windows one on MC2 and another Abuntu one on MC3. Uh so give those about 30 seconds or so. Those will be started up and start running as I mean obviously something happened. they've went fully offline, but they will start up without any intervention at all. And that's kind of the whole point of this process. And there we go. We're all running less than two minutes for that entire thing.
So here at the bottom, you can see MC5 is completely offline. And more importantly, there's no VMs that are associated with it. That means if let's say MC5 was just knocked offline because of a network interruption, it's not going to reestablish connection with the cluster and start its VMs again. those VMs have now been completely reassigned to new nodes. Now, obviously, if you have a node crash, you probably still want to have a tech take a look at it, but at the very least, this gets the VMs back up and running, and they'll start running through any automated scripts you have set up and hopefully bring your services back online with fairly minimal interruption.
So, wrapping this all up, the cool thing about servers, networking, PCs, there isn't one correct way to put everything together. There are best practices for securing your data and making sure your services stay running on the servers that you run, but there's also a million ways to achieve the results that you're after. One of the reasons a lot of people decide to run HomeLabs is to tinker with different solutions. And the cool thing about what I showed off with Proxmox today is it doesn't need $40,000 worth of hardware to get started with it. A couple years ago, I built a similar Proxmox high availability cluster running a trio of $300 servers and demoed the exact same high availability configuration. The concepts and settings for this can work whether your budget is $500 or $50,000. So, if you've ever wanted to try running clustered storage and virtualization setups, there's no better time to try and get started.
Thanks again to Super Micro and Solidine for lending me some amazing hardware in the MicroCloud node server along with the PS1010 Gen 5 NVME drives for this project. If you're interested in learning more about the hardware used in this video, I will have links down in the video description. I'll also have written instructions for creating a Proxmox cluster available down below. On your way down there, make sure to drop this video a like and subscribe to Craft Computing if you haven't done so already. Follow me on the social medias at Craft Computing for daily shenanigans like this. And if you like the content you see on this channel and want to help support me in what I do, consider joining the Patreon. Link is also down in the video description. You can also help support the channel by buying some of my merch. All of my merch was designed and is made right here in house by me. We've got pint glasses, whiskey stones, rocks, glasses, stone coasters, bottle openers, a whole bunch of stuff that is actually cut on my laser right next to my server rack. So, if you want some truly unique and awesome quality merch, craftcomputing.store. And that's going to do it for me in this one. Thank you all so much for watching and as always, I will see you in the next video. Cheers everyone.
Beer for today is an annual tradition. It is a collaboration between Fort George Brewing, Mirage Beer, and Sun River Brewing this year. This is the three-way IPA. They come out with this every single year, and it's different every single year. Uh it's usually a collaboration with a couple of other Oregon breweries. Sometimes they expand beyond that, but uh always a fun one to get a hold of. I love that even though you're cooperating and being on the desk, you are still just barely off camera. Yeah, just enough to let people know you're there.
So, the Fort George 2025 3-way IPA, 7%. Um, I'm normally a really big fan of this one. I don't think I am this year. There there's a couple of things that I'm genuinely not liking about this year's three-way IPA. And this is hard for me because I think there's some really good notes in here. It's just there's a couple extremes that are pushing me to to not really enjoy this one. It's trying to be super super bright, but it's bright in like a vegetital sense. It It's It's that West Coast IPA that's more celery and grass than it is floral or citrusy. And so it's got that grassy taste just cranked up to 11 right off the bat, but then it's also super super dry throughout the entire palette. While I can see what they were kind of going for here, that is like a a bright summery IPA that's nice and refreshing, this one makes me more thirsty. It is so dry. It It's gone from crisp to grating if I am explaining that properly. Yeah, it's like celery and crab grass cranked up to 11 right off the bat and then it kind of settles into this dank but still very vegetital like mid thick section. But then as soon as you like swallow the drink, it completely sucks all of the moisture out of your mouth and leaves you with this overly bitter flavor. I think this is just just a miss for me. I I think it's still probably a fine beer. It's just a couple of flavors that can be a little grading on me. They're here in their strongest presence that I've ever had them in an IPA. Not one I'd try again. But if those flavors that I described are ones that are in your wheelhouse, you're probably going to love this one. So, different people have different preferences for flavor, and that's totally okay. I hope you enjoy this one. Say goodbye, Rambo. Say goodbye, buddy. Yeah, you're internet famous. Did you know that?