5 Uses for NVMe Storage in your Home Lab Server
I really think storage is an often overlooked aspect of setting up and configuring a home lab server. Most know about CPUs and memory as super important when running virtual machines and containers. However, storage, is equally important. In my honest opinion, if you aren’t running NVMe drives in your home lab server, you are missing out and potentially shooting yourself in the foot in your learning endeavors. NVMe is a game changer for learning and running home labs. Why? Let’s see why you need NVMe in your home lab server and explore this further.
Table of contents
What is NVMe?
First of all, what is NVMe? NVMe stands for Non-Volatile Memory Express. It is a high-performance storage protocol that is specifically designed for solid state drives (SSDs). The key to NVMe is that unlike traditional storage interfaces like SATA or SAS, NVMe takes advantage of the PCIe bus directly, which allows it to have a much lower latency and higher bandwidth compared to the traditional storage interfaces.
It supports parallel processing and multiple queues. This enables SSDs to handles thousands of requests that are simultaneous. For home lab enthusiasts, this is what you want. NVMe drastically improves performance in data-intensive applications, specifically virtualization. NVMe comes in different form factors, including M.2 (arguably the most popular), U.2, and PCIe add-in cards.
Why you need NVMe in the home lab
There are five reasons we will cover as to why you need NVMe in the home lab. Let’s look at those reasons now:
- It has very cheap performance for the price
- It handles random IOPs very well
- It allows running leaner virtual machines
- It enhances the performance of traditional storage
- It can be used as memory
Let’s consider these one-by-one and how these advantages of NVMe are totally worth it in the home lab.
1. It has very cheap performance for the price
There is no question that NVMe has very cheap performance for the price. You can get Samsung 990 Pro NVMe drives for the following at the time of this writing (below are Amazon affiliate links that give me a small commission). These give you Seq. Read Speeds Up to 7,450 MB/s.
- Samsung 990 Pro NVMe 1TB drive for – $109
- Samsung 990 Pro NVMe 2TB drive for – $167
- Samsung 990 Pro NVMe 4TB drive for – $318
You can also get Gen5 NVMe drives now as well on Amazon and other places for fairly cheap and the performance is mind-blowing, up to 14,500 MB/s.
- Crucial T705 2TB PCIe Gen5 NVMe M.2 SSD
- SABRENT Rocket 5 1TB Advanced Performance Internal M.2 PCIe GEN 5
- Corsair MP700 Elite 1TB PCIe 5.0 x4 NVMe M.2 SSD – Up to 10,000MB/sec
I remember back in the day working with an Equallogic PS6510 SAN with 24 or so 10,000 RPM spindles and the unit maxed out at around 14000 IOPs. It is amazing to think a little NVMe “bubble gum stick” size hard disk has just as much performance in the Gen5 PCIe version.
2. NVMe handles random IOPs very well
One of the distinct advantages of NVMe compared to traditional storage is that it handles random IOPs very well. When you look at random I/O patterns of 4k or so, these are some of the most common patterns in the world of virtualized workloads.
This is the strength of NVMe. So, it makes sense that for a home lab, using NVMe storage would provide tremendous benefits for performance to satisfy this use case. When you have many virtual machines running, these tend to generate lots of random 4k IOPs load on the storage subsystem. This is well-suited for NVMe storage technology.
3. It allows running “leaner” virtual machines
What do I mean by this? This is something that I have discovered over the past few years of running a home lab primarily off of NVMe storage. What I have found is that you can get away with running a leaner virtual machine than you might be able to do historically with slower storage. NVMe helps to mask RAM-starved machines.
When a machine is RAM starved, it means that you will typically have more paging operations to disk. This will bring traditional storage to a stand still. However, when running NVMe backed datastores for running your VMs, you will find that even when you have leaner VMs from a RAM perspective, these will operate and perform well if these are on fast storage such as storage backed by NVMe devices.
This is a two-fold benefit. Number one, you can get by with less memory, which in the home lab is often the most strained resource, especially if you are using mini PCs. Number 2, you will be able to run more VMs with less hardware, which is a double benefit.
4. It enhances the performance of traditional storage
What do we mean by the next point? Specifically when dealing with NAS devices and NAS operating systems like TrueNAS. You can stick NVMe in front of traditional storage as a cache tier. This allows you to cache reads and writes first to your NVMe storage, and then these IOPs are passed along to the underlying traditional storage.
This gives you the advantage of having the capacity benefit of traditional magnetic disks which are now huge and the performance of NVMe. So your traditional storage “feels” like fast SSD storage, even though it really isn’t. This can allow you to run workloads on NAS devices with traditional spindle storage that you wouldn’t be able to do otherwise, at least at an acceptable level of performance.
5. It can be used as memory
When you have NVMe storage in your home lab, you can use NVMe as a type of “memory.” With VMware vSphere 8.0 Update 3, VMware introduced the new NVMe memory tiering feature. This feature allows you to designate an NVMe device as a “tier” of system memory, even showing up on your ESXi host as such. You can configure the percentage of your current physical RAM that you want to allocate to NVMe memory tiering.
Check out my video on the topic below:
Also, recently, I ran a very fun experiment where I took Intel Optane NVMe storage and used this as a insanely fast Swap storage for my Proxmox host. By doing this, I was able to have the feel of a Proxmox host with much more memory that I had. It is yet another way that you can benefit from NVMe storage in the home lab.
Wrapping up
NVMe storage is one of the major developments in the home lab over the past few years that has allowed most of us to take our home lab experience to the next level. NVMe is fast, its getting much larger, and it is relatively cheap for the amount of performance you get. These 5 use cases and benefits of having NVMe storage in the home lab hopefully sheds light on the utility value of this ultra fast storage technology. Keep your eyes peeled for deals on NVMe storage, such as the ones we have linked above.
