Transcription
Hey everybody, it's Christian, and in this video, I want to reveal some of my best practices and config settings for Proxmox. Proxmox, as you should know, is a really nice hypervisor platform. I'm a big fan, and I've been running this since the very early days of my home lab. And over time, I've collected a list of all the important settings I'm configuring on any new Proxmox installation, or things that I'll do whenever I need to install a new virtual machine. Honestly, I believe you should never run Proxmox without knowing these settings. Yeah, it might sound a little dramatic, but I'm sure you will agree that there are some pretty important topics you should always take care of, such as keeping your Proxmox server up to date, or configuring storage and backups. And I hope this will help you to make all your Proxmox setups more secure and robust, while getting the most performance out of your hardware.
But before we do that, we quickly need to talk about another important topic, which is how to secure the remote access to your home lab services. And I want to point you once more to one of my absolute favorite solutions called Twin Gate. And a big thanks goes out to Twin Gate for sponsoring this video. Twin Gate is a zero trust network access platform that is easy to deploy, even easier to use, and always secure. In my previous video, we've already covered the essential benefits of the ZTNA concept in it, why it is so much better than using VPNs, and how to install and set up Twin Gate in your home lab completely for free. And Twin Gate just recently added a couple of new and very exciting features I just want to highlight here, such as usage-based auto-lock, which automatically revokes the access to inactive accounts, aerial access that can expire credentials, DNS filtering, and client configs to enforce security policies, and a brand new Kubernetes operator to quickly and easily implement ZTNA to your Kubernetes environment. You can be sure, of course, I will cover those new and cool features in an upcoming video. But also, if you want to get an easy and secure access to all your home lab services, Twin Gate is really amazing for that. You can use it completely for free up to five users and 10 different remote networks. And it runs on your NAS, Raspberry Pi, in the cloud, and of course, in a VM on your local Proxmox environment. So check out Twin Gate, start making your network more secure and safe. Of course, I will leave you a link to their website in the description box down below.
Okay guys, so here I'm logged into my Proxmox environment at home. As you can see, I just recently built a new Proxmox cluster with two nodes. By the way, if you want to check out the tutorial, I'll leave you a link in the description box down below. But regardless of your setup, all these best practices and tips we're going through here in this video are always working on any Proxmox environment. So it can be a single node cluster, it can be a two or three node cluster, it does not matter at all. And I'm configuring those things on any Proxmox server that I deploy in my home lab.
And the first item on this list, and this is probably by far the most important thing that you should take care of when running Proxmox, is that you configure the update procedure correctly. And the reason for this is, if you go to the repository section, you can see that by default, there are one or two entries in that list here that start with Enterprise. And those Enterprise repositories, you can only receive updates from them if you've got a valid subscription. And now, what I always do is, because I never buy a subscription for my home lab environment, I simply just deactivate those Enterprise repositories. So make sure to disable the Enterprise environments and then add a new one. And from this repository list, you can just choose "no-subscription". And if you want to have the Ceph repository for distributed storage and a three-node cluster, you can also enable the Ceph Quincy or Reef repository with "no-subscription". So this will add a new item to that list that you can also enable or disable, that will just download any of those updates for Proxmox even without a subscription. So note, you will still get this annoying notification. But when you have done that, you can simply go to Updates, Refresh, and then it will automatically check all the active repositories if there are new updates for Proxmox.
And now, in that list here, you can just go to any of your Proxmox nodes and click on "Upgrade". By the way, if that is grayed out, that is the reason because I'm logged in using my authentic uh IDP provider. Now, if you switch back to the standard authentication and log in with a root user, you can just open any of your Proxmox nodes, go to Updates, and click on "Upgrade". This you just need to confirm, and then it will pull down the latest upgrades for your Proxmox environment. The same of course also works if you log into the shell with the root user, execute an `apt update` and `apt dist-upgrade`, and it will also pull down all the latest upgrades for Proxmox and install any new services automatically. So when you've configured the update repository for Proxmox correctly, you should just upgrade your environment from time to time. For example, I do that every one or two weeks on all of my Proxmox nodes in the cluster. And of course, sometimes they also need a reboot, so I'm also rebooting those nodes from time to time to apply those latest kernel updates.
Okay guys, so the next item on my list is to enable notifications. This is also pretty important to know what's going on in your environment. So to do that, we need to go to Data Center and scroll down, down in that list to Notifications. So here you can see there is a default entry that will just try to send a mail to the root user's address, which doesn't get anywhere. So it's really important to add a new notification target. So you can use the send mail or the SMTP uh sending process. I would always go and recommend SMTP because this allows you to configure a custom email server. And once you've done that, it's also important that you go to Notification Matcher. So you can also create a a new one, or you just modify the existing notification matches. And here you could also specify any match rules and so on. Usually, you don't need to do anything. You just need to enable the targets to notify and then switch from the default mail to root and activate your custom notification target that you have configured with your SMTP credentials for your administrator email address. Uh, once you have done that, the notification system should be active and running, and you could use it for not getting notified about any backup jobs, for example.
But before we do that, we need to go through another important item on my list, or at least in my opinion, it's pretty important, and that is issuing a trusted TLS certificate for your Proxmox server. The reason why I think this is important, yeah, because it's just best practice in IT and it allows you to get rid of this annoying certificate warning in your browser whenever you want to open a web connection to your Proxmox server. Now, if you want to issue a trusted TLS certificate, you should know there are a few things you should have in place before doing this. So you should have a public domain registered somewhere. For example, I've registered the domain `sealcreative.de` that I'm using. And you should use a DNS provider, for example, Cloudflare. Cloudflare is a great example because it's completely free and you can use it to issue Let's Encrypt certificates using the DNS challenge, which is pretty simple to configure. I will show you that in a second. And what you should also have is a DNS name or a DNS host that you configure on your DNS server that resolves the host name of your Proxmox server to the internal IP address. So, for example, in my home lab, I'm running a DNS server. If you have watched my BIND DNS tutorial, you will probably know what I'm talking about. And this will automatically resolve the DNS names, for example, `prox-production1.home.sealcreative.de` to the internal IP address of my Proxmox server. The same is also working for the second node. So this is just a second DNS entry that resolves the second node to its IP address. Now, you can use a local DNS server for that, but of course, you could also just enter the Proxmox addresses with the local IP address on the Cloudflare DNS provider, then it's publicly resolvable, but actually doesn't really matter. So there are two possible ways how you can configure this. Just remember, you will need a public domain and a DNS provider, something like Cloudflare.
All right, so once you have done that, you need to go to Data Center and go to ACME, which is the ACME service protocol that allows you to automatically issue and also renew certificates using Let's Encrypt. And I personally just like the DNS challenge the most because it's the easiest one. It doesn't require a connection from the Let's Encrypt validation server to your local environment. It just works with a DNS provider and an API token. Super simple and easy. Just go to Challenge Plugins, click on "Add", and then just give it a name, something like "Cloudflare DNS", and select your DNS provider. So make sure you select yours. And in the case of Cloudflare, you will need to authenticate with an email address and an API token. You could also use a password in here, but I think the token authentication is probably the most secure and the easiest one to create. For example, on my Cloudflare account, you can see there are all the public domains that I've registered and configured here. So I just need to go to my profile section on the left side, go to API Tokens, and then you can simply create a new API token. Make sure you enter the API token in here and also add your administrative email address in here. And then you need to go to Accounts. Here also add a name, your email address again, and then select the ACME directory. So make sure it's selected, the first one, which is the production environment. You could also, if you're using that for testing, you're not right sure if it works for you, you could also select the staging environment. But just know that the staging environment issues certificates that are never trusted. These are just for testing. And if you move to a production environment, make sure it's using this directory here. Accept the terms of service and click on "Register". Then you should have two entries here, one for Accounts and one for Challenge Plugins.
So once you have that, you can go to your Proxmox environment. So you need to do that for every node in the cluster and go to Certificates. So here you can add your ACME certificates. Make sure that it's using the account. You can also edit this here and select the Cloudflare account. And then add the Subject Alternative Names, the DNS name that resolves to the internal IP address of your Proxmox environment. Make sure it selected the DNS Challenge and not HTTP, because, as I said, HTTP challenge requires a connection from Let's Encrypt to your internal Proxmox environment. Select your plugin that you have previously configured, and then just add the domain name for your Proxmox environment. In my case, `prox-production1.home.sealcreative.de` that resolves to the internal IP address. By the way, you can also add more Subject Alternative Names. So if you want to use multiple DNS names for your internal IP address, for example, you can see that I have added for the node 1 this DNS name, but also `px-cluster1`. And on the second node, it looks similar. It also has the `prox-production2` for the node identifier, but also the same `px-cluster1` name. And the reason why I'm doing this is, if you have watched my last video about the Proxmox cluster, you know that you can manage any VM from any node in the cluster. And I simply created a load balancer object that will forward the incoming traffic on this domain name to any of those two nodes depending on which one is currently active. So I don't need to remember which node is currently active and running. I can just use the `px-cluster1` name and it will automatically forward the traffic to any of the nodes. Now, if you don't add this name to the Subject Alternative Name for both nodes, you would see a certificate warning when you're doing this because the name would not match these certificates. So that's why the cluster DNS name should also be part of the certificate. So just add, add multiple names, as many as you want, as many as you need. And then click on "Order Certificates" now. This will start the ACME certificate issuing process. If you have done everything correctly, it will take a few seconds, order new certificates, and then you should get rid of the certificate warning in your browser.
All right, so that's it about certificates. Let's go and talk about storage. Storage is also a pretty important thing you should definitely take care of when configuring Proxmox. And that is done in the Data Center Storage menu, just as simple as that. And the reason why I'm showing you this is that I always configure the storage for my Proxmox server depending on if I'm adding multiple drives, if I want to add redundancy to that, and if I'm configuring a target location for a backup. This is also really important. And the weird thing in Proxmox is that if you configure storage, you also need to add a content for the storage. And that defines what Proxmox is able to store on that storage location. So you can see by default, you should have those two entries: "local" and "local-lvm". While the first one will store the backup files, ISO images, container templates, and only the second one will store the disk images or containers, so the actual drives for the virtual machines and containers.
One thing that I always do on my Proxmox environment is configuring an NFS drive. And this is pretty important. For example, this one here is an NFS drive that I have enabled on my NAS storage. So here on TrueNAS, I have a storage pool and a specific dataset that I've created just for storing virtual machine backups. So that allows me to store my virtual machine backups outside of the file system of Proxmox, which is much more secure if the Proxmox server dies or if any local drives on the Proxmox server are damaged or whatsoever. I have those backups for the virtual machine somewhere on a different location, on a different file system. Even though the NAS is currently running as a virtual machine in my Proxmox environment, I still use it for storing. Here you can see I've added this storage location to this IP address, I've mounted this specific path, and I've also enabled it for the content VZ dump backup file, so that I can store backups on this external file system. And then you can access this NFS storage from any node in the cluster. You can see Proxmox 1 has access to this storage, and this is where my backup files are stored. As well as Node 2 has access to it. This is by the way also a possibility to add a shared storage for all the VMs if you want high availability in a two-node cluster, but that's a topic for a future video maybe.
All right, yeah, let's talk about backups. Backups is also one of the most important things that you should always run on Proxmox. You should never run a VM without backups, at least a production virtual machine. So that's why you should definitely set up a backup job here in the Data Center Backup menu. So you can see I've added a backup job in here for some of my virtual machines. You can enable all the VMs that should be included in the backup, and depending on which node they are running. For example, as you can see, I've added only production systems, excluded my NAS storage, of course, because this is a target location, so I don't need to store this VM there. But for any of my demo environments, I'm not using backups because, yeah, those environments are just temporary testing stuff. I could easily create recreate them without a backup. But any of your nodes that you want to backup recurrently, just enable. And then you can set a schedule. Also, don't forget to set the target storage, otherwise Proxmox will try to store that locally, and that usually doesn't have much disk space. So that's why it is important to configure an additional storage location for taking backups. Now, make sure this is selected here, and then add a schedule. For example, I'm just taking a backup every day. And then it's also important, if you have paid attention to step two and configured the notification system, that you should switch the default auto system to the notification system here. So whenever it's taking a backup, you will receive an email if it's successful or not. And if the backup job is not running, you should definitely pay attention to it and repair it. So it's important to make sure this is set to notification system here. Also, the compression mode, you can just leave it by default. The Zstd is pretty good by default. And also the snapshot mode is uh important. If you want to continue running these VMs, then a snapshot job is definitely the best one because it doesn't require to shut the VMs down in order to take it back up. It's automatically doing this while the VMs are still running. It's pretty cool. And yeah, then you should also set up the retention. So that tells Proxmox how many backup files it should keep on the system, because if you're just a smaller file system, this file system might run out of disk space pretty soon if you're taking a job every day. So therefore, I've configured to keep only the last 10 backup files on the NFS storage. I could also shrink it down a bit, but I think it's pretty important because if you're losing data, you probably don't notice it the same day. You might just recognize it three or four days later. And therefore, it's important to keep a few more than just the last backup file, so that you can go back in time and find the day where you want to restore the files from.
All right, so that's it about the backup, and I think this is really a must-have for a good Proxmox environment.
Okay, so next item on my list is PCI passthrough. I think this is also a pretty important topic, especially if you want to virtualize file storage systems and you want to pass through graphics cards or storage controllers, maybe even network cards to some of your virtual machines. So there are a couple of things that you should have enabled, for example, IOMMU, which is important for PCI passthrough. And yeah, this requires a CPU that supports IOMMU, and you also need to enable this in the BIOS. So on most modern motherboards, it should be enabled by default. And then you can simply just check on the Proxmox system if it has IOMMU enabled. So just go to one of your Proxmox servers, execute this, and then you should see "IOMMU is enabled". I could also do the same on my first node, which is an AMD system, and you will see a slightly different result. So here it says "IOMMU version 2 AMD is loaded and initialized". Also check if the remapping is enabled. This is the result on the Intel CPU, and also this is the message on the AMD system. And then you can enable PCIe passthrough. Let me also show you how that looks like, for example, on the second Proxmox node. I said I virtualized my TrueNAS storage server. And the way that works is, if you go to the hardware section of that VM, you can see there is a PCI device that passes through the internal SATA controller on the motherboard to the VM. So all the drives that are connected on the motherboard to the SATA ports are not accessible on the Proxmox server itself. They are only visible to the virtual machine that is running there. Now, maybe we should clarify if that's a good idea to virtualize a storage server. As I said, if you're passing through the storage controller, there's no real difference than running it on bare metal, actually. So there is no real reason why you could or should not do that. And yeah, it might be pretty useful for some people that don't want to install a separate hardware just for adding storage.
But also apart from the PCI passthrough configuration, I also created a list that I always go through when I create any new virtual machines, because some of those default settings on Proxmox aren't really the best. So let's go through some of the best practices for creating VMs as well. And let's select my Proxmox uh node 2 and create a new virtual machine here. I'm just entering "test VM" and let's go through these settings one by one. So first of all, you should pay attention to the guest operating system that you're installing. So when you're picking a Linux distribution, you will see the type "Linux" in here and also the newest kernel that is enabled by default. But if you're using a Windows system, you should definitely make sure that you're switching this to "Microsoft Windows", select the specific version, and also enable this checkbox "Add additional drive for VirtIO drivers". And on Linux, those VirtIO drivers are automatically built into the OS, so you don't need to install them. But on Windows, they are not part of the Windows operating system, and that's why you have to download the Windows VirtIO drivers from the Proxmox website, and then you will get an ISO file that you will need to upload to Proxmox. So here, just make sure the VirtIO ISO is selected, so that you have that ready when you're installing the operating system. By the way, here you will also see a screenshot what you need to do in the installation process of a Windows guest operating system.
Okay, but yeah, in case you are using Linux, you just click on "Next" and you should pay attention to those next settings here. So first, make sure the SCSI controller is set to "VirtIO SCSI". You can use the single mode, that's totally fine. And you always should enable the Guest Agent, the QEMU Guest Agent. This makes sure that the guest operating system is better integrated into the Proxmox virtual environment, and Proxmox or the host OS has access to some internal functions and metrics of the guest OS, such as the balloon driver for memory, and also the IP address and MAC address of the guest OS. In case of Windows, you should also add a TMP and add a TMP storage. Again, this is required on modern Windows operating systems. On Linux, you can ignore this. When you're using a graphics card or you're using a Windows OS with a graphical UI, you might also go and select the graphics card and switch it to "VirtIO" or "GPU", which makes sure it has a better performance when rendering graphical applications in the VM. Now, on any other system that doesn't have a graphics card, just leave it by default, that's totally fine. And also, on most of my Linux servers, I just use the default machine, the `i44fx`, and the default BIOS. So those versions are the older legacy versions that are better supported on older systems. But again, if you have a Windows OS or if you want to do PCI passthrough, for PCI passthrough, it's required to select the machine to `q35`, which is the newer version that supports PCI Express and newer functionality, as well as the BIOS, which you can set to OVMF, which supports UEFI features. Now, again, only a setting that I've configured on my NAS storage server, because there I needed PCI passthrough. For most of the other operating systems, I haven't enabled this, except for Windows, where the UEFI features are pretty useful.
But also network, make sure that when you're creating a virtual network interface, that you are selecting the VirtIO model. So on some older operating systems where it might not be supported, you can also switch to a different one, like the E1000, is the I, I think the oldest driver that is around, or the Realtek one for, yeah, any compatible settings, or the VM one if you're importing virtual machines from VMware that might not have the VirtIO drivers. But in most cases, the VirtIO drivers are just the best for Proxmox and the Linux OS.
All right, so that's basically everything you need to pay attention to when you create a new virtual machine. I hope you made some notes here. But yeah, I'll do one more extra tip for you. Let's do one more extra tip, because what is also pretty cool is that you can create VM templates. And that is what I always do. For example, I've created a VM template for my Ubuntu virtual machine. And that allows me to easier and simpler provision new virtual machines with a cloud-in-drive where you can install a new VM. You clone it from this template, but you can also set a new user, a new password, give it a new, upload a new SSH key, give it a different IP address, and so on, without setting or configuring those things on the VM guest operating system. It's pretty cool. Now, you can create those templates in here, basically, if you have installed a VM that you have prepared. So you probably should have deleted any SSH keys or uh, reseted the machine ID and so on. I'm not going into details here, but you can click on "Convert to template" and then store it as a template to clone a different VM from it. It's much simpler than always going through the installation process of a guest OS. But of course, I've also made a video tutorial about that topic and used another tool for creating those templates automatically in a scripted way with the HashiCorp Packer application. It's so super useful. And also, I'm using any other systems in combination to Proxmox like Terraform, Authentik for my authentication. So there are so many more things, but I will cover them in additional and upcoming videos.
Okay everybody, so I hope these settings and best practices helped you to make your Proxmox setup more robust and secure. For me personally, these are just must-haves. And if you knew all these tricks and you think I missed a few important ones, then please leave me a comment down below so I can add them to my best practice list. And as always, thanks everybody for watching. A big thanks goes out to all of the Patreon supporters. You guys make all of these free tutorials possible. And yeah, everyone, have a nice day. I'm going to catch you in the next video. Take care, bye-bye.