Transcription
What's up guys? Welcome or welcome back to the channel.
In today's video, we're going to dive into Proxmox storage. Our servers are up and running. Our network is configured and now it's time to tackle one of the most important parts before we start building, and that is storage. I try to make these videos as detailed as possible, which means that they can be a little bit long, but like I always say, you do not have to watch the whole video. I have included chapters in the description box below. So, please feel free to jump to whatever part interests you the most. With that being said, let's go ahead. Let's get the video started.
In Proxmox, storage is managed in two levels, just like networking. You have the data center level and then, of course, you have the node level. The node level is where you will work directly with physical disks. This is where you will create and format storage types like ZFS pool, LVM groups, or directory storage. These are local to the node, not unless you share them manually.
The data center level is for centralized management. It is where you will map existing storage and define what the storage is going to be used for. So, VM, ISO, or backup. You can also set permissions, and this is also where you will mark whether or not you want that storage to be shared, or you want it to be node-specific. So, you don't really create storage at the data center level; you just reference what has already been created at the node level. So, think of it this way: the node level is what's physically available, and the data center level is what's used and who can use it.
So, what we're going to do is we're going to start at the node level because this is where we define what storage is physically available and how it's prepared. But before we break down the different storage types at the node level, I think it's important to understand the difference between a file system and a storage type because they're not the same. And I think that understanding that difference will give us a really good understanding of how things work.
So, a file system is how data is actually stored and organized on the disk. So, think of ext4, xfs, zfs, or btrfs. A storage type is how Proxmox interacts with that storage. It defines how things like VM backup and ISO are managed. It's the structure or the interface, but it's not the format. Okay.
Okay, so now we can talk about the different storage types at the node level. And number one on the list is directory, which is used as a regular folder on a mounted file system like ext4 or xfs. You basically use it to store VM disk, ISO, or backup as a file. It is simple and it is very flexible.
Number two is LVM, which creates logical volumes from physical disks. It offers good performance but lacks advanced features like thin provisioning or snapshots.
Number three on the list is LVM thin, which is a more flexible version of LVM that supports thin provisioning and fast snapshots. It is also pretty great for VM storage.
And last on the list, of course, is ZFS, which is a powerful file system with built-in volume management, snapshots, compression, data integrity, and so much more. It is ideal for advanced setups, but it's also ideal if you want to push your storage a little bit more in Proxmox.
Okay, so now that we have a really good understanding of the different storage types at the node level, let's go ahead. Let's take a look.
So, for us to be able to configure our node-level storage, we want to go to the node that we have set up in our data center. So, we're going to go to Nodes, and in this case, we only have one node, which is the node zero node, right? So, we're going to click on that. And then in there, we're going to see something that says storage. So, we have we're going to see something that says disk, not storage. So, if we click on disk, it's going to show us all of the disks that we have available on our server. And as we go down the list, you're going to see all of the storage types that are available at the node level. So, you have LVM, you have LVM thin, and then you have directory, and then, of course, you have ZFS. So, we're going to walk away from top down. So, we're going to start with LVM and then work our way all the way down to ZFS.
So, LVM, like I said earlier, LVM is kind of like a logical volume that is created from the physical drive, right? So, to create an LVM in Proxmox, and we're going to create it, and we'll talk about them a little bit more as we go along. To create an LVM, all you have to do is just click on this guy and then click on this button right here that says Create Volume Group. And then once you click on that, it's going to ask you which one of your disks do you want to use. Then I can say, hey, I want to use number one. And um, it's going to ask us what do you want to call this? For now, I'm going to call this LVM test, because we're going to create VMs and attach VMs to the storage. So, we can test a couple of things. So, for now, I'm naming them according to the storage type, so we can be able to quickly attach them to our VMs as we create them. So, as you can see, we have LVM test. And here you have the Add Storage. Um, so what that means is that if I check this box, this is automatically going to add this storage to our data center, and we're going to check that in a minute when once we start talking about the data center level. So, we're going to go ahead and check the box and then we're going to go ahead and click Create. It's going to go ahead and create the LVM storage. So, as you can see, the LVM volume has been created. It's telling us that the size is 4TB. That is perfect.
So, the next on the list is LVM thin. So, like I said earlier, LVM thin is kind of a more flexible version of LVM that supports thin provisioning and fast snapshots. So, we're going to go ahead and create it, and I want to talk about the thin part of things a little bit more so we can get a better understanding of it. So, to create an LVM thin storage, we just click on that. And then we're going to go ahead and click, same same concept, same process. We're going to go ahead and click on Create Thin Pool. And then again, it's going to ask us what storage do you want to use. We're going to select one of our storage. And then we're going to call this LVM thin test. And then we're going to say, okay, Add to Storage again. So, this can be added to the data center level. And the goal here is that you can either add this to the data center level when you create and when you go through the creating process for each and every one of the storage, or you can manually add it at the data center level if you don't have this checkbox checked. But for now, we're going to go ahead and check that and click Create.
Now, one of the things to keep in mind, and one of the things that I've noticed is that with LVM thin, it takes a little bit longer to create than all the other storage types. So, if you do click create and it seems like it is taking a little bit longer, that is perfectly fine. So, we're going to go ahead and click Create, and I'm going to let it do its thing.
So, let's talk a little bit about the whole concept of thin provisioning. So, the thin provisioning side of things basically just means that you can assign your VM more storage than you physically have. So, you can basically overprovision, but you cannot overuse the physical space. So, you cannot overuse the 3.97TB that you have here. So, for example, you can create five VMs and give all of them 5TB storage, right? And they will all work perfectly fine with no issue. And that's because the VMs only use the disk space they actually write to, not the full amount they are allocated. So, if you give your VM 5TB, they're not going to use all 5TB of the storage. They're only going to use what the VM currently needs to run. Now, this is different from other storage types because for other storage types, if I was to give my VM 5TB, that 5TB is physically taken from that drive, from the total of this size space that we have here, and you're not going to have that amount of space to allocate to other VMs anymore. But within provisioning, you can create as many VMs as you want. You can give them all the same size or more than the size that you see here. But the only downside is that you, of course, which I don't know if it's a downside, that it makes perfect sense, you cannot overuse what you have here. So, if the VMs start going above the 3.97TB, then you're going to start receiving some errors. But if those VMs are going to stay below that, they only get to use what they need, and you're going to create as much VMs as you need to. It's a beautiful thing if you have a very small environment, right? Because then you can create as many VMs as you want and assign them as much space as you want. You just have to keep an eye on your space, and those VMs can use the space that they need as they start growing. So, that's basically what a thin provision is.
Next up is directory. So, a directory is basically just a folder, a regular folder that is mounted on a specific file system. So, to create one, we're going to go ahead and just click on Create Directory here. It's going to ask us, it's going to pre-select the next disk that's in the line, or you can choose whatever disk you want to use. And then it's going to say, what file system do you want to use? You can use ext4 or xfs. Ext4 is kind of like the default, the industry de facto. XFS is mostly used for like larger storage types like backup and things like that. So, I'm going to choose the first one on the list here. And then we can call this whatever we want. So, for now, we're going to call it dir dir. And then we're going to say, yes, we want this to be added to our data center. And then we're going to go ahead and click on Create. So, it's going to take a couple of minutes, and it's going to go ahead and create that. The beauty of having a directory is that with a directory, you can store a lot more stuff in there because it's file-level storage. So, basically what that means is that with a file-level storage, you can store a lot more stuff in there. So, backup, ISO images, and all this other cool stuff in there. With a block-level storage, you can only store like the VM disk or the container disk in there. You can't store anything outside of that, and we'll take a look at that in a minute. So, a directory gives you a lot more flexibility to be able to store a lot more stuff in it.
Next up, we're going to create a ZFS storage. ZFS is both a powerful file system and a volume manager with a lot of built-in features. So, to create a ZFS storage, all we're going to do is click on ZFS, click on Create ZFS storage. And then we're going to call this whatever you want to call it. So, in my case, I'm going to say ZFS test. And here you have a couple of options. So, as you can see, it has a lot more options than the other storage types. So, here you have the RAID level. You can select whatever RAID level you want to choose. I'm not going to go over this because I don't want to make the videos too long. I have talked about the different RAID levels and what they mean in my previous video. So, you can take a look at that in the true video. I kind of break it down. So, if you do want a clear understanding of what each RAID level means, you can go ahead and check out that video. And then you have for now, actually, I'm going to just choose single disk just for testing. And then you have the compression type. So, ZFS basically compresses data on write and then it decompresses it on read automatically. So, here you can choose the compression type that you want. So, in my case, I'm going to choose LZ4 because it's lightweight and it's fast. And then the ashift. Ashift basically tells ZFS how to interact with your drive. So, 12 means that it's 4K, and you can go lower if you want to if you have like a spinning disk. So, this all depends on the type of disk that you have. In my case, I have a SAS, so I'm just going to select 12 for now. Here you get to choose what disk you want to be included in your RAID. Since I'm doing single disk, I'm just going to choose one. But in your case, depending on what RAID level you choose, you can choose whatever disk you want to include in your RAID. Again, you have the checkbox here that says Add Storage. If you check this box, right, it's going to add the ZFS storage to the data center. We're going to take a look at that in a minute, and then we're going to click Create.
Now, one of the things to keep in mind about ZFS when you create a ZFS storage, or about a ZFS storage, is that a ZFS storage can act both as a block-level storage and as a file-level storage. Now, remember when we talked about the directory storage, we said that directory storage is a file-level storage, and because of that, it allows us to be able to store a lot more stuff inside of it, and we'll take a look at that in a second. Now, a ZFS storage allows us to be able to do both. You can choose for it to be a block-level storage, or you can set it up so it's a file-level storage. Now, to do that, what we want to do is that if you were to click on Create, when you check this box, when you name your ZFS, and you have this box checked, what is happening is that when you have the Add Storage box checked, Proxmox automatically adds it to the data center as a block storage. This type of storage is only used for things like VM disk and container.
So, let's go over to Storage here. So, we're going to go to Data Center and we're going to go to Storage to take a look, to try to understand what I'm trying to say. Right? So, we're going to look at each of the storage. Now, these two we're going to skip because the local ones are the ones that I initially created when we created Proxmox. Right? So, we're going to skip those two. Now, if we go to Directory, you can see that in our directory storage that we created, that we created, have a lot more options. We have backup, disk, image, ISO, container snippets, container templates, and all of this other stuff that we can add to it. So, you can click on all of this, you can upload an ISO, you can do a lot more stuff to it, and that's because it's a file-level storage.
Now, if we were to go to LVM, the LVM, one of the LVM tests and LVM thin, we can see that all we can upload to these guys are VM disk and container volume. Outside of that, we cannot upload anything to it. That's because these are block-level storage. Now, if we were to go to our ZFS, you see that that's the same thing. And the reason for that is that when we initially created our ZFS, if we go back to our ZFS storage and we click on Create ZFS, we have this box checked, right? So, when you have, again, when you have this box checked, Proxmox automatically adds it to the data center level as a block storage. So, when it adds it as a block storage, all we can do is add VM image and container volume to it.
Now, if we want to create a ZFS storage and we want it to act as a file-level storage, what we want to do is we want to click on Create, and we want to uncheck this box. Then we can call this whatever we want. So, we can call it ZFS_DIR. And then make sure this box is unchecked. And you can do something like mirror. And then you can check two here because in a mirror setup, in a RAID mirror level, you need two. And then you can do whatever you want here. Your compression level. And of course, we're going to keep that as 12. And then we're going to say Create. Right. So, we're going to give it a second to create this. Now, once it's done, we're going to see it right here.
Now, the next thing that we want to do is we want to find out the mount point for this guy right here. Because again, what we're going to do essentially is that we are going to create a ZFS data set at the node level, which is what we just did right here, and then we're going to add it manually as a directory storage at the data center level. This basically lets Proxmox use it for storing files like ISO, backup, and templates, like we would on the directory storage. So, we want to find out what the mount point is for this guy right here. So, to do that, we're going to go to Shell and we're going to clear, and we're going to say ZFS list. And this is going to list all of our ZFS storage. And as you can see right here, this is the one that we just created. And we can see that the mount point right here is this guy right here. So, we're going to take this, right? And we're going to go back to Node Storage, cuz I um disk, right? Uh, we're going to go to uh ZFS and ZFS directory. If you look over here, it's not listed here yet.
So, now what we're going to do is we're going to go to Node. We're going to go to Data Center, and then we're going to go to where is it? Disk Storage. We're going to go to Storage and we're going to click on Add. And we're going to click on Directory. And then what we're going to do, we can call this whatever we want. ZFS_DIR. And then it's going to ask us what directory do you want to mount this on. We're going to choose our mount point. So, our mount point again is ZFSDIR. And then we're going to say we want to have this content, all of this content to be able to upload all of the things inside of this directory. So, now we're going to click on Add, and then it's going to appear right here. So, now, as you can see, and it's still doing its thing. We're going to give it a couple of seconds for it to finish.
Now, as you can see, it's finished. It doesn't have the question mark next to it anymore. Now, as you can see, we've created a ZFS storage as a file-level storage. And because of that, now we're able to add all of this things in it. The beauty of this is that now we have a ZFS storage with all the ZFS features, and now we're able to add a lot more stuff inside of it compared to this guy that we initially created, and it was created as kind of like a block-level storage, and we're not able to add a lot of stuff into it.
Okay, so let's talk about ZFS for a few. So, if you mount a ZFS data set as a directory in Proxmox, like we just did, you can use it for VM storage, but one of the things to keep in mind is that you will not be able to take Proxmox-managed snapshots of those VMs, which is a drawback to consider when you're doing something like that. When it comes to choosing the storage type that you need, so whether you want to choose ZFS, LVM, LVM thin, or directory, you have to carefully think about your resources and your needs. ZFS offers data integrity, compression, built-in snapshots, and a lot more other features. But the thing is that it is very heavy on RAM. If you have limited resources, or especially memory, you might want to skip ZFS and do something like LVM thin, which is lightweight. It supports snapshots and it uses space efficiently, making it a good option for smaller environments. However, if you have enough resources, you could combine ZFS block storage for VM disks with a ZFS directory for ISOs and backups. Or you could use something like LVM thin for VM disks and a directory on ZFS for other storage needs. In the end, you have to pick what best fits your setup and your priorities. You might have to make some sacrifices depending on your resources and needs. But ultimately, you have to sit down and think about what you're trying to build and then choose the storage type that's best for you.
Okay. So, what we're going to do is we are going to create a VM real quick. So, I can attach LVM to it. So, I can show you guys some of the things I was talking about where I said that you cannot create a snapshot with LVM and you can with LVM thin. Um, so, what we want to do is we want to go ahead and create create a VM here on the top, and then we're going to call this whatever we want. So, we can call this test, right? And then we can source pool. We can just do test lab. Click next. And then here we can say we want the local because that's where our ISOs are right now. And then Pop OS. We can leave this as default. Click next. And we can leave this as default. And go ahead and click next. And then for the disk, we're going to select this guy CSI. And then we're going to say for the storage type. So, now if we click on the dropdown, we can see all of the storage that we have set up. We have uh these are the local ones. We have a directory, LVM test, LVM thin test, ZFS directory, and then ZFS test. We're going to do LVM test and then we're going to leave the rest as default, and you can make this whatever you want. So, you can do 100 if you want, or whatever storage you want. Okay. So, this is this is a good place to talk about the difference between LVM and LVM thin. I know I've touched on it, but with LVM, for example, if it's 100, if I have about 100 gig on the main storage and assign 100 gig, all of that 100 gig will be will be used. However, um if uh with LVM thin, if I have like 100 gig, I can assign as much as I want here, and the only thing that's going to be used is basically the stuff that's being used by the OS. So, I hope that that helps a little bit. You guys understand the difference between LVM and LVM thin. So, I'm going to go ahead and click next. Uh, leave the CPU as default. Doesn't really matter. Click next. Network is fine. Um, let's go ahead and click next and then just click finish.
So, we're going to go ahead and see virtual machines here. And now we have one of our virtual machines that we're running test on. So, we have hardware. If we go to disk here, you can see that we're using this guy right here, LVM test. And now, if we go to uh where is it? Where is the snapshot? You can see that we cannot take snapshots here. We cannot take a snapshot because again, with LVM, you cannot take a snapshot. But with LVM thin, you can take a snapshot. And if we go to backup here, for example, and we click backup, you can see all of the things that are available here for backup are the things that have. If we go back here real quick, I'm going to click and X out of that. If we go to our storage here real quick, and we go to let's say one of our directory, so this guy, you can see that we have image, we have backup, right? These are the file storage or the storage, not the file storage, the storage types that are able to contain things like backup. So, if we go back to our node and then we click on, sorry, if we go back to our VMs and we click on our VM here and we go to backup and it says backup. Now we can use any one of those directories to be able to back this VM up if we want to. And of course, if we have something like Proxmox Server or Proxmox Backup Server set up, we'll be able to back our VMs to that as well. So, that's one of the things that I wanted to show you. You're not able to take any sort of snapshots whatsoever. So, if we go to snapshot one more time, see that we're not able to. But if I was to clone this, for example, right? Let's say clone. And I say the storage type that I want to use is LVM thin test. And we can call this whatever we want. Test two. Resource pool. We're going to do test lab again. And then we're going to click clone. And we let it do its thing.
So, once the VM is done provisioning, so what you want to do, you can do something like take a snapshot. And then we can write whatever here and then write whatever here and then take a snapshot. And you can see the snapshot. We're now able to take a snapshot. So, that's the main difference between a thin provision and LVM thin provision and a regular LVM. And also, that's of course the way you attach the storage that we just created to all of your VMs as well.
All right. So, now let's move on to the data center level. So, like I mentioned earlier, you don't create storage at the data center level. You pretty much just map storage that was either created at the node level, or you're bringing in an external storage from outside into Proxmox. This is also where you would define what the storage will be used for. Some of the local storage that we already covered. So, things like directory, LVM, LVM thin, NBT, RFS can all be mapped at the data center level. You also see external storage options like NFS, SMB. So, basically anything that you bring in from outside into Proxmox will typically get mapped at the data center level. So, you can share it across all of your nodes in your data center. So, that's one thing to keep in mind as well. Now, we're not going to dive too deep into external storage just yet. I don't have much of it set up for this environment, but as I build things out and learn more, I'll definitely bring you guys along for the ride. With that being said, let's go ahead and take a look at how Proxmox handles storage at the data center level.
Next thing we want to talk about is storage at the data center level. How storage is handled at the data center level. So, to do that, we want to get away from the node and we want to go to the data center level. And once we click on that, of course, we're going to see storage. And then here you're going to see this option right here that says Add. So, one of the things that you're going to notice is that you're going to see all of the things that we've added when we were setting up things at the node level. So, and of course, the local ones as well, right? So, we have the directory, LVM test, LVM thin test, and then we have ZFS directory, and then ZFS test. Again, the whole idea of the data center level storage is to one, map things that are already internal, or to bring things from the outside and then bring them into Proxmox. And to do that, all you have to do is basically just click here, and then you can see all of the available things that you can do here. And of course, you have a lot more here because again, it's both for internal use and external use. And when I say internal, I'm talking about your nodes, right? So, all of the nodes underneath your node tab here, you can map storage from those nodes, or you can bring things in.
So, for directory, for example, we just did a directory for the ZFS directory that we just set up. So, you call it whatever you want. You'll put the directory here, like we did for ZFS. And then after that, you can do things like enable it. Right? If you check on enable, it will tell you that that drive is enabled. Right? And then here you can select the node. In my case, I only have one node. So, if you have more than one node, you can select that. And of course, if you click Share, it means that more than one node can act, more than one node can access the storage. And of course, you can select, as I said before, it's also where you will select what exactly you want the storage to be used for in terms of the content that you want to be able to upload to the storage. Just like we did for the ZFS directory, then you can check all of this as to what you want to be uploaded to the storage. Again, just mapping things that already exist at the node level.
So, if you go down the list, I'm going to X out of that. If you go down the list, you have LVM. Same concept. If I hadn't mapped the LVMs that we created, I will call this whatever I want. So, it will be something like test. And then it will tell you like, hey, you know, based base storage existing volume group. And then it will tell you like LVM test and LVM thin test. You can select one of these. And of course, you only have two options here because again, this is kind of a block storage. And then you can select what you want. And then here you select the node, right? And then you will select enable. You select whether or not you need, you want it to be shared. That means that one node can access it. And then here it has the wipe removed volume. Basically, what that means is that if you check this box, when you delete things from the data center level, it's also going to wipe whatever is available at the node level. So, just be careful when you check this box. If you don't necessarily want to wipe things at the node level, make sure this box is not checked. But if you check it, it's going to wipe things when you delete things at the data center level, it's going to wipe things at the node level.
And if you go down the list again, BTRFS, again, same concept here, you just map things. And then NFS, and again, as I said, the data center level is not too exciting like the node level, because all you're doing is just mapping things. And SMB, right? So, if you have an SMB server, then you want to map the drive that you have set up there in Proxmox. And that's something I'm going to do once I have TrueNAS back up and running. Going to set that up to be able to map that in here. So, maybe I might create a video about that. And you can choose, of course, what you want to, what you want the content, or what content do you want to be able to upload in the storage, and then you can put the server and all those sort of things in here and be able to map things. So, X out of that. And then you have CFS, RBD, and then you have ZFS and ICE. Again, same exact concept. The data center level, I know I sound like a broken record, it's mostly for mapping things, right? So, you're just mapping things that already exist in your environment, or you're mapping things that are coming in from outside of your environment. And then, of course, you have ZFS. You can map your ZFS stuff here. All right. And it gives you the ZFS pools that are available. I guess this is one where you can check your, this is one where if you don't want to do a ZFS list, you can come in here and check the ZFS pools that you have available as well.
And then here you can do Proxmox Backup. So, if you do want to add a Proxmox Backup Server and you don't want to back up things to your local storage, you want to back up things to your Proxmox server. Proxmox server is something that I am going to set up. This server is actually going to serve two purposes, the server that we're on right now. One is going to be my testing environment, and also it's going to be my kind of like my storage and backup environment. All right. So, that's something I'm going to set up here as well. So, you can put the server, the ID, username, and password, and all this sort of stuff. Again, this goes back to bringing things from outside and then bringing them in. And then last but not least, you have ESXi. So, you can connect if you have an ESXi area, you can connect that and be able to pull things in from there as well. So, again, not too exciting at the data center level, but the data center level is pretty straightforward. All you're doing is you're mapping things. So, you can map whatever it is that you want to map here. It gives you that centralized control to be able to share things across all of your nodes. And it makes it easier to manage at one level instead of just managing things at the node level, because you can only imagine if you have several nodes in here. You don't want to go to every single node to be able to manage your storage. You want a central place that you can just go to and be able to set up storage or break down storage or configure storage or share data among storage and all that stuff. And this here, storage at the data center level, allows you to be able to do that.
All right. So, that is pretty much it for this video, guys. We are in a good spot to start building things out. Now, the servers are up and running. The network is solid, and now our storage is ready and good to go. Next, we're going to start setting up the core components of the environment. So, in the next video, we'll be spinning up our DNS server and getting to understand DNS a little bit more. So, don't forget to like and subscribe if you like the content. Stay geeking, and as always, I will see you guys on the next one.