VMware vSAN Home Lab 10 gig Network Switch Upgrade
Like any virtualized architecture, the network is an extremely integral part of VMware vSAN. With the software-defined architecture of vSAN, all storage activities take place across the network for the most part. If you are running an all-flash vSAN environment, 10 gig is definitely the way to go. In the home lab now for a couple of years, I have been running the Netgear X708T, 8 port switch. However, I was in need of additional port count for 10 gig and also was looking for additional features. In this post, I want to cover my VMware vSAN Home Lab vSAN 10 gig network switch upgrade, the replacement model I chose, features, and other considerations.
Reasons for upgrading the 10 gig network switch
The Netgear X708T has served me fairly well over the past couple of years. I haven’t had any major complaints with the switch and stability has been good with no issues with switch crashes or other complaints.
However, since I had added a fourth and fifth host into my home lab environment, I have exhausted the available ports with the X708T. Also, there were a few things that I have never liked about the Netgear.
For one, it is a smart managed switch that basically limits you to configuring and administering the switch through a web UI. For what I have wanted to do for the most part, this was ok. I simply am using the switch as a simple layer 2 10 gig switch that aggregates my 10 gig links for vSAN purposes.
I have 5 or 6 VLANs running on the switch. However, I was starting to feel a bit limited with the features of the switch and management options.
Additionally, for my four SYS-5028D-TN4T Servers, these have 10 gig Base-T connectivity and the X708T switch has 8 ports for this purpose. The switch also has (2) SFP ports as well. However, if you uplink either of the SFP ports, it is an either/or use of the (2) SFP ports vs port 7 & 8 of the 10G-Base-T ports.
Essentially, you are losing two ports for uplink purposes. If you are using dedicated NICs for VMkernel port traffic for vSAN, you don’t really need to uplink the switch. However, if you are using NIOC and you are also using your NICs for 10 gig traffic for VM port groups, you will need to uplink your switch for ingress/egress traffic for those virtual machines.
VMware vSAN Home Lab 10 gig Network Switch Upgrade
I looked at various switches, however, I had my eye on a couple of different switches from Ubiquiti.
- US-16-XG Unifi Switch
- EdgeSwitch 16 XG
Below, you can see the similarities between the two switches. They physically look identical. The Unifi US-16-XG sports the typical Ubiquiti white chassis, which I really like the looks of.
The EdgeSwitch 16 XG is a black matte finish chassis. However, the real difference between the two comes in how you can manage the EdgeSwitch.
The EdgeSwitch 16 XG can be managed as a standalone switch from the command line, web interface, etc. I chose the EdgeSwitch 16 XG for that reason.
While I like the Unifi platform, having a vSAN switch that I couldn’t really manage as a standalone unit made me decide to go with the EdgeSwitch. Also, I found the EdgeSwitch was as cheap as the Unifi US-16-XG.
As you can see from the picture, the EdgeSwitch 16 XG has (12) SFP ports and (4) 10GBase-T ports. What I like about the switch as well is that all ports can be active. You don’t lose ports by uplinking certain ones.
As a summary of the great features of the switch. It allows for the following:
- 160 Gbps total throughput
- 12 SFP+ ports
- 4 10GBase-T ports
- MSTP/RSTP/STP
- VLAN
- Private VLAN
- Voice VLAN
- Link Aggregation
- DHCP Snooping
- IGMP Snooping
- TACACS+
- RADIUS
- 802.1X
- MAC Filtering
- ACL
- DiffServ
- CoS
- Static Routing
- DHCP Server Functionality
Setting up the Ubiquiti EdgeSwitch 16 XG
I really like the flexibility of this model in terms of setup. You can attach a console cable to the unit and configure from the command line. You can administer it from the web interface as well. When it boots, it attempts to reach out to DHCP for a dynamic IP address. If it doesn’t find a DHCP server, it configures the default IP on the unit which is 192.168.1.2.
The default credentials are:
- user – ubnt
- pass – ubnt
The first thing you will want to do is login and get your ubnt default password changed. In addition, you will read some lengthy threads about issues with Ubiquiti switches without upgrading the firmware. So, the first thing I did out of the gate was to get the firmware to the latest and greatest version which at the time of this writing is v1.9.2.
Ubiquiti makes finding new firmware and downloading very easy. I can generally hit the correct page on the first Google without even having to visit the Ubiquiti site and trying to figure out where the switch firmware is located.
Once you have the firmware downloaded, you can simply login to the web interface of the EdgeSwitch 16 XG switch and click the little “cog” underneath the System actions area of the settings page. This is the update firmware option.
The only thing I don’t like is that the page just spins without any visual cues that the firmware is uploading, installing, etc. You just have to wait patiently for the firmware to update. For me, this took probably around 5 minutes of waiting.
You will eventually get a prompt that the firmware has been updated and you need to reboot the switch.
Current EdgeSwitch 16 XG Firmware issue
One issue that I found with the current firmware and it looks like it has been an issue with previous firmware releases has to do with enabling jumbo frames. In the web interface, when you enable jumbo frames, it just toggles on and when you save it, it toggles back off.
There is a current Ubiquiti thread on this issue you can find here:
However, I found that when I enable jumbo frames from the command line interface, it seems to stick and even vSAN does not flag the “large packet size” in the vSAN health test, so apparently, the command line configuration is working, despite not showing that jumbo frames is enabled in the web UI. This is something I will update this post on if this is fixed in the future.
Navigate to the configure prompt, select your interface range, and set the MTU.
The command to enable jumbo frames:
- configure
- interface 0/1-0/16
- mtu 9216
How do you confirm that jumbo frames are enabled?
- show ip interface <your int number>
Great Ubiquiti EdgeSwitch 16 XG Features
Despite the current firmware bug that is not allowing the configuration of jumbo frames from the web UI (a great reason to have a switch that you can manage at the command line level), the switch is performing beautifully so far. I really like the new UI that has been introduced as well.
One really great little feature of the new Dashboard in the UI is the view of real-time bandwidth for each port which details how traffic is flowing. I kicked off a storage vMotion between my vSAN cluster and a standalone host just to generate traffic and give the switch some work to do.
Below, you can see the UI gives you a very detailed view of the port traffic (ingress/egress direction). You also get a total throughput detail on the switch. As you can see I am barely generating any traffic compared to the total throughput the switch is capable of.
Integration with the Ubiquiti UNMS Cloud environment
In case you haven’t heard, the Ubiquiti UNMS cloud service is now free. UNMS Cloud is a cloud-hosted environment that allows you to configure, upgrade, and monitor several different Ubiquiti platforms: airMAX, airCube, airFiber, EdgeMAX and UFiber. Unifi devices are managed by the UniFi Network Controller.
Signing up for a UNMS Cloud account is now free. Once you have signed up to create your UNMS account, you can then add your switch to the UNMS cloud.
As you can see below, you just click the +UNMS in the upper right-hand corner and populate with your UNMS cloud key.
Wrapping Up and Impressions
So far, the VMware vSAN Home Lab 10 gig Network Switch Upgrade using the Ubiquiti EdgeSwitch 16 XG switch has been a worthy upgrade and I am very pleased with the overall performance and features of the switch.
It works great with vSAN so far and I haven’t had any issues with network traffic outside of the current bug in the firmware that does not allow configuring jumbo frames properly without the command line.
I found the overall value the switch provides hard to match by any other switches on the market for this price range. If you are looking for a good vSAN 10 gig switch, the Ubiquiti EdgeSwitch 16 XG is a great one to look at.
