VMware NVMe Memory Tiering Questions Answered (Memory over NVMe)

I have memory tiering enabled on a second MS-01 I bought just for this purpose.

I also discovered that svMotions fail without much explanation. Storage migration for a powered-off VM works fine.

There may or may not be an issue with a Docker instance I have running on an Ubuntu Server VM. Within Docker, I have an installation of Dashy, which is the dashboard I use for most things here. Dashy is set to display when I open a new tab in my browser.

After a couple of days I've noticed, repeatedly, the Ubuntu VM OR Docker OR Dashy becoming less and less responsive, sometimes to the point where it doesn't load and I have to manually refresh the screen to get it to do so.

I don't know in which component this is happening or what's causing it. As of now, I have moved the Ubuntu VM (with Docker and Dashy) back to the MS-01 that doesn't have memory tiering enabled, and the VM has been fine for several days. I will eventually move it back to the memory tiered machine to see if the issue recurs.

Yesterday, I powered on for patching an infrequently used secondary vCenter appliance on the memory tiered machine. I will watch how this fairly large and complex VM behaves on this machine. I already have one vCenter running on the machine, and it seems fine.

I think there's a bit of development work still to go on the memory tiering/M.2 thing. One feature would be tiering of large memory pages. Another would be partitioning just a portion of the M.2 for tiering and leaving the rest of it either for the ESXi installation or for use as a datastore. Additional development might allow vMotion/svMotion of certain VMs to/from memory tiered machines that we can't currently with the preview. This includes VMs that have memory reservations.

I am not sure that memory tiering in ESXi is the "game changer" for home labs that I saw it called. But it might turn out to be a handy thing to have on a -- non-critical -- virtual machine host that is typically short on memory but still has processing power to spare.

@jnew1213 Great observations! And glad to know someone else making use of it. I definitely see A LOT of potential with this technology, but as mentioned, there are a lot of items that are still quite a bit rough around the edges. 

It will be great to see how it develops, but definitely lots of features that need to be added or "turned on" with this technology. It makes me wonder if VMware already has the ability to do svmotions, etc but just not enabled in this tech preview likely. I also like the idea of being able to use a "partial" device partition for tiering so you can take advantage of the rest of the drive as well. But wondering if they will ever do that since caching devices are I guess best practice best to be dedicated to that task?

I did have a weird failure on my MS-01 with vCenter the other day (my main home lab vcenter) where I woke up and monitoring was dinging me about all VMs being down, well it was just vcenter, but console had CPU unrecoverable errors on the console. A quick reboot fixed the issue. Not sure if this is related to NVMe tiering, or if it is a byproduct of the hybrid CPU architecture that caused an issue like that. 

Let me know if you see anything similar to that as well

I had a weird issue yesterday that I've been reminded of while reading your last post.

A network cable was severed (by a rabbit) yesterday morning and that brought down a number of things including my Plex server. Replacing the cable restored everything, or so I thought.

A friend emailed me late yesterday that Plex seemed to be down. I checked and, sure enough, the VM had hung on restarting at a point before the operating system loaded. Powering the VM off and on again quickly fixed the problem.

I've never seen a VM hang where it did before, at the looking for network boot phase.

This particular VM, Windows 11, is on the MS-01 with memory tiering in effect, and has an Intel ARC 310 GPU passed through to it.

As I said, I haven't seen this kind of hang before, but I'll be keeping my eyes open for it happening again going forward.

Ah! One other thing that needs to be built into memory tiering: M.2 lifecycle management. Currently we don't know how much life any M.2 used by ESXi has left. (iDRAC, etc. has an estimate on properly initialized datacenter class devices.) It's probably less of an issue when used as a datastore, but when used as "memory" an M.2 that spontaneously fails can possibly corrupt a lot of stuff!

@jnew1213 Wow that is an interesting story you will have to tell me at some point. I have had a lot of things happen with network cable but never a rabbit Definitely keep us posted if you see any more VM hanging behavior. I do think with the technical preview (as we know it will be) things are not quite as stable with tiering turned on.

Also, that is another great point about lifecycle management for M.2 devices, especially for m.2 devices used for tiering. These do become super critical at that point, so a failure needs to be handled carefully. I am sure these are probably things their engineering team is working through hopefully. Will be cool to see how this feature evolves. 

I do know too, as I mentioned in the first list, you can mark multiple devices as tier devices. I didn't find any specific guidance or details on this as of yet, but wondering if both devices can be used for redundancy possibly? More questions I guess!