Perfect Home Lab 2024: MS-01s, NVMe Memory Tiering, and NVMe NAS
I think I may have found my perfect solution for home lab at least at this time. I now have two MS-01s in a cluster configuration, with NVMe memory tiering enabled and an NVMe NAS. Let me give you guys the run down on why I went this route and how everything is working out so far.
Table of contents
My journey to the perfect home lab in 2024
I have been talking about this on various videos showcasing my home lab since 2022, into 2023, and now in 2024. I have been searching for “THE” solution to replace my aging Supermicro Xeon-D units that have served me well now for the past several years. The time has come to find something newer and more robust from a processing standpoint and efficiency perspective.
When the MS-01 came out, and I did my review of the unit in January of this year, I knew that we had something special in this unit. It has quickly caught on as the home lab mini PC that everyone has been using in their home labs. It really checks all the boxes of things that I was looking for to replace my Xeon-D with just a few exceptions to that. But, even if it is not perfect, it is almost perfect as I noted in my review.
Mini PCs are extremely powerful and appealing in 2024
When it comes to hardware to power your home lab, the ultimate is enterprise server gear. However, most of us can’t afford to go out and buy a new Dell PowerEdge or other variety server, and what’s more the energy costs of running an enterprise server can eat you alive over the course of time if you are running them 24x7x365.
This for me has become a realization that I have accepted as I think about what my perfect home lab looks like in 2024. I thought about replacing my aged Supermicros with newer Supermicros, but honestly, I just didn’t see the point of the investment in that kind of gear for a home lab with great options like the MS-01. A few years back, these were the only option in my opinion to have a truly capable home lab, but that is no longer the case.
You can read my review of the Minisforum MS-01 mini workstation here:
VMware vs Proxmox: VMware wins with NVMe memory tiering
While so many may disagree with this, I still think VMware has the best offering for home lab environments now with the new NVMe memory tiering feature. This is just something that Proxmox doesn’t have currently, but could add in the future. What is it?
With a simple software configuration in VMware, you can add “memory” to your system by way of a dedicated NVMe drive in your system. VMware ESXi intelligently tiers memory between the fastest (DRAM) and the slower (NVMe) storage.
It will keep “hot” pages in DRAM, and then memory that isn’t needed as quickly in the NVMe tiered memory storage. So, this allows me to now take a mini PC that maxes out at 96 GB of DDR5 memory and now take that up to 478 GB of memory! Pretty dang cool.
This provides the best value IMHO for home labbers looking to extend their mini PC hardware to run more workloads (VMs and containers, etc).
Below is a quick screenshot from setting up a local NVMe device as the NVMe memory tiering device.
Below is a look at one of my cluster hosts on the Configure > Hardware > Overview menu, you will get an overview of the memory tiering and the various tiers, amounts, etc.
You can read the official VMware blog here: vSphere Memory Tiering – Tech Preview in vSphere 8.0U3 – VMware Cloud Foundation (VCF) Blog.
Moving from HCI to traditional shared storage
One of the interesting pivots I am seeing in the enterprise with organizations that I have been working with is that many are leaving the idea of vSAN HCI in the VMware world in favor of traditional storage now. The reason is, you guessed it, Broadcom.
Broadcom’s licensing changes that now charge you for vSAN by the terabyte and this is going to add up for many customers. So I am seeing several nix the idea of going HCI with vSAN and instead doing traditional storage.
Also, another reason I am going the shared storage route is that I need one of the NVMe slots in the MS-01 dedicated to NVMe memory tiering, which limits my options when it comes to HCI configurations. This would mean at most I would need to do the original storage architecture (OSA) with vSAN and not the new enhanced storage architecture (ESA).
The MS-01 only has (3) NVMe slots. So one of these will be a boot drive, one will be an NVMe memory tiering device, so it is best to go the shared storage route with iSCSI LUNs. I could do vSAN without NVMe memory tiering, but I think this is a major win for home labs and I definitely want to take advantage of it.
This is the route I am following at least for now with this build of home lab that I will run for the next while. I am using an NVMe NAS device that will contribute the shared storage for my two Minisforum MS-01 mini PCs.
You can read my review of the Terramaster F8 SSD Plus here:
You can watch my YouTube video review of the unit here, including performance of the unit:
Don’t get me wrong, I love HCI and especially love vSAN as I have a lot of miles behind me with it in the home lab especially and it has not let me down, has been very very resilient to failures and also has been easy to manage. I am not saying I won’t go back to vSAN in the future, but for now the shared storage between the two MS-01’s is an appealing option.
The bill of materials (BOM)
While many refer to production environments with traditional storage as a 3-2-1 with (3) hosts, (2) storage switches, and (1) SAN, my setup is effectively a 2-1-1 with (2) hosts, (1) network switch and (1) SAN/NAS.
Let’s look at the bill of materials (BOM) for this iteration of my home lab refresh:
- (2) Minisforum MS-01 mini workstations
- Core i9-13900H (14 processors)
- 96 GB of DDR5 memory (478.66 GB with NVMe tiering)
- Connected via 10GbE
- Added the kernel boot mod for dissimilar processors
- Booting each from small NVMe drive
- Each has a 1TB NVMe drive for NVMe memory tiering
- Each has another local 1TB NVMe drive for a local datastore
- (1) Terramaster F8 SSD Plus NVMe NAS
- (6) 2 TB Samsung EVO 980 Pros
- Running TRAID (hybrid RAID5 + RAID1)
- Connected via 10GbE
- (1) Ubiquiti Edgeswitch 16XG
- 16 10GbE ports
How is it working out?
I just got everything put together and it is working out really well so far. I am pleased with the performance of the configuration. The MS-01’s with the Core i9-13900H processors are wildly faster than my Supermicros that I am coming from with the Xeon-Ds.
Also, the little Terramaster F8 SSD Plus with all NVMe storage is pushing around 70,000 IOPs and the management features of the NAS are good so far, no complaints there. I think for now, this is the perfect home lab server and storage configuration for me in the 3rd quarter of 2024.
Let me know if you have any questions in the comments about this setup or what your idea of a perfect home lab is in 2024, as I may not have covered everything. But this will be a living post as I will keep things updated here on how I have things configured.
Thanks again Brandon – a really interesting post.
It would be interesting to have some figures though to understand how this “NVMe tiering” compares to using just plain old swap. That would help understand if VMWare really has that advantage relative to Proxmox, or if it’s more marketing fluff.
Michael,
Thanks for the comment as always! This would be an interesting comparison for sure. From what I have seen the NVMe tiering is more intelligent, but would still be good to compare. Are you full on Proxmox on the home lab or still running ESXi?
Brandon
You are missing the bill part of the bill of materials. How much did this come to and where did you find the best deals?
Dennis,
Good point. Part of my hesitation on the actual pricing is that the pricing I don’t think has been officially set on the Terramaster NAS that is officially released this month. However, I believe it is around $799 for the Plus model. Also, the MS-01s are around $679 and cheaper with discount codes that are often floating around.
Thanks again!
Brandon
I’d love to see an article doing a deep dive on NVMe memory tiering (which NVMe drive are you using, performance and latency evaluation) and a comparison to other options (e.g., swap on NVMe, tweaks to swappines, Optane as swap).
Michael,
Thank you for the comment! Check out my other site where I went into more detail on how this is configured step-by-step: https://tek2cloud.com/nvme-memory-tiering-in-vsphere-8-0-u3-quadruple-your-mini-pc-memory/.
Brandon
Today I’m full on Proxmox, but “magically having more RAM” is really tempting.
So if NVMe tiering can be backed up by figures I’d definitely use ESXi also – hell, I’d probably even go with vCenter so I can use it with Terraform or other IaC tool.
In either case I’d opt rather for the Minisforum MS-A1 – obviously the MS-01 is amazing hardware – but I still prefer to go with MS-A1 only needing 2.5Gpbs networking today.
Thank you for your article. I have a question to the MS-01. Some Users are reporting that the MS-01 continuously crashes or reboots. Mostly reported by Proxmox user but also from ESXi users. Do you have see those crashes too?
Alex,
Thank you for the comment. I can confirm that I have not seen any crashes with the MS-01. I have been running a single MS-01 as a standalone ESXi host in the home lab since the beginning of the year and then recently have added another MS-01 into a vSphere cluster and no crashes. I have upgraded memory to 96GB and running the ESXi boot parameter to allow the dissimilar cores in the CPU. Not stability issues so far.
Brandon