My home lab upgrades this 2023

My home lab is a bit scattered so I had to turn some parts into a collage.

Top: "Main Cluster" - 5-node ProxMox setup running nearly all house services. Has capacity for experiments but each node can run everything by itself if I need to go into low-power mode. The Synology NAS next to these nodes provides NFS shared storage for the nodes and is isolated from the house data.

Middle: "Sandbox" - 5-node ProxMox cluster running a number of VMs to support Kubernetes experiments. It's a work in progress, 5 litres of tiny-mini-micro power. Each node has a short link to the dedicated switch. Video goes through an HDMI switch for easy node selection. Power goes through a Kasa 6-in-1 power switch with energy monitoring so that I can cycle nodes remotely and watch/log power usage. 

Bottom: "OG NAS" - First built in 2014, this system is an absolute trooper. 4x 6TB HDDs and dual NICs, this is the primary copy of house data. 

"New NAS" - Also a work in progress, this is aiming to replace the aging OG-NAS. 5x 6TB HDDs, 64GB RAM, i7-4590K and a pair of 16TB drives for snapshots and additional copies. This is a ProxMox server running TrueNAS in a VM (HBA for drives passed through). It hosts NFS and SMB shares for the house data and other various projects that haven't been migrated off the OG-NAS. 

Total Idle power draw of everything pictured is about 300W. 

Heh, I blame the Raspberry Pi shortage, really digging into ProxMox and eBay for starting that trend this year. 

The main cluster is a mix of Dell, Lenovo and HP tiny/mini/micro nodes. (One Dell is on my bench for service so only 4 are pictured today) The Dells and Lenovo are Core i7-8700 (6c 12t) processors, the HP nodes are AMD Ryzen 5 3400G (4c 8t) and all have 32GB of RAM, 512GB NVMe storage and a 1TB SATA SSD for shared storage. (Sync'd ZFS pool in ProxMox)

The sandbox cluster was my "oops" cluster. I sent a low offer to a seller with a Lot of 5 thinking there was no way they'd accept, and they did. I then asked my wife for forgiveness. 🙃 They are a homogenous collection of Dell Optiplex 7050s with a Core i5-6500T, 256GB NVMe storage and 32GB RAM each. In almost every case, I took advantage of the default build config where Dell likes to stuff a single stick of memory. I was able to pick up matching pairs to get 32GB without having many sticks left over.

@malcolm-r @ghaleon @t3hbeowulf check out this potential home lab server....Aoostar NAS with Ryzen 5800U, 6 NVMe, 6 HDD, 10 gig, coming in January, looks to also be barebones if I am reading correctly: https://www.virtualizationhowto.com/community/home-lab-forum/aoostar-nas-with-ryzen-5800u-6-nvme-6-hdd-and-10-gig-network-home-server

The short answer: I'm going to stick with my current hardware for a while longer because it is MORE than capable of running everything I throw at it.

The longer answer: With the exception of storage, almost everything I've built was purchased used. I have a soft guideline that I don't upgrade unless the equipment is experiencing some sort of imminent or current failure, or I have backed myself into a corner of storage/capacity and need to build out a replacement. 
It all started with a single "NAS+Desktop" that was pictured at the bottom of the collage in my photos post. A Core i3-3220 dual core processor, 16GB of RAM. 
It "works", but QuickSync on that processor is slow and 2 cores was constrained for running Portainer and a number of containers in addition to Samba for the file share. 

I started pricing out replacements when I stumbled upon the micro PCs. I picked up one, then two for "redundancy" and started to get familiar with ProxMox. I picked up a pair of micros from HP because they had Ryzen processors in them and I wanted to toy with passing through the integrated graphics to a VM. Since I had 4 nodes and "even numbers are bad", I picked up a 5th and settled there. Any single node in the cluster is enough to run every service I have, but I spread them out over several nodes to ensure migration/backups keep working. (I test migration frequently) Of course, this didn't solve the original "NAS" problem but it did solve the workload/capacity problem. 

The NAS in my picture was created because I had just enough other spare parts laying around that I thought: "with a used motherboard and some drives, I can build a new NAS". So I did... and have been slowly migrating storage volumes over to it, while keeping the trusty old Core i3 system online for now. 
I'm finally in a place where I don't have any spare parts itching to find a use and a homelab complex enough to make hardware purchases an extremely considered action. I'm considering consolidation and using a more disciplined approach to spreading services out over nodes, and also staying off eBay. 

I have this task pinned.  It's one of my higher priority goals for 2024... better documentation.