vmbr0 without physical uplink

@life-from-scratch, thank you for signing up for the forums! Glad to help through this one. Just curious, can you post some screenshots of the virtual interface configuration and if any VLANs are in play? Also, just a reminder to post some pics and specs in the thread here (just reply to that thread) for a chance to win a mini PC. 👍 

Currently everything is pretty much default. I'm assuming that bottom interface that PfSense shows is the vmbr0 from Proxmox, the other four are on my PCI card. Proxmox network settings are absolutely stock aside from the static IP I gave it. eno1 is the motherboard nic. 

@life-from-scratch, one thing you can do to begin with is for sure identify which network adapter is being used in pfSense for the connection to vmbr0 in Proxmox. You can take a look at your NIC assigned in pfSense and see which mac address lines up with what you are seeing on the network adapter in Proxmox. Let me know what you find there.

Yes, the MAC address shown in Proxmox matches that last interface shown in PfSense. 

@life-from-scratch. It sounds like you just have a flat single VLAN network. Is that true? I think this should be pretty easy to configure, but it looks like where things are getting complicated is plugging the devices directly into your Proxmox host. What you need is a small network switch to aggregate your devices as things will probably be difficult to make work as expected plugging into the host directly. I know, especially with the small space for your lab, you are probably wanting to have as few cables as possible, but I think you will always be fighting unexpected behaviors trying to set it up this way. The Proxmox bridge isn't really designed to be a network switch in itself for physical resources. I think you would like to have a network switch with VLAN capabilities in the future as well if you want to delve into carving up your traffic even further for different traffic types, etc.

Ideally, to simplify, you could just VLAN-aware enable the bridge, and have all 4 connections backing the bridge. Then you can plug your devices into a VLAN aware switch and plumb traffic as you need from there. Let me know if you would like to continue to experiment though without a switch. I could probably set up a test host and play around with configurations from that angle.

Yes, no VLANs yet. I have a small switch already. I've messed with wiring don't remember exactly what's connected where at the moment (see cable management lol). It all works fine, I just literally have a cable with one end plugged into the network card that's passed through to PfSense and the other end plugged into the motherboard of the same machine. It just seems like since PfSense and all my VM's are connected to the virtual bridge that that physical cable shouldn't be needed. I would still have the PfSense LAN port going out to the switch.

Ah ok gotcha. Passthrough will change things a bit. With passthrough, that VM will have exclusive access to that physical device. So it makes sense that you would need to somehow tie this into your vmbr0 bridge. Without passthrough, you could have pfSense and Proxmox be able to communicate "in the box." Here is a way you could set that up.

With a few VLANs, you could carve up that traffic so that everything is switched virtually inside Proxmox between pfSense, your Proxmox host, and everything else. 

As an example, you could create:

VLAN 100 - WAN 

VLAN 200 - LAN

VLAN 300 - other traffic if needed

That way, you could plumb everything in on the pfSense side virtually. You could have a virtual interface in pfSense tagged with VLAN 100 for WAN, a second virtual interface tagged with VLAN 200 for LAN traffic, and an optional interface tagged with VLAN 300. With this configuration, you could have a single cable out of your Proxmox host into your switch on a trunk port, and then have your devices plugged into access ports tagged with the appropriate VLANs if that makes sense.

I think I understand, but not really. Why did it not work to simply set up a bridge in PfSense between the virtual interface from Proxmox and LAN? In PfSense I can make one of my physical ports into WAN and set the remaining three physical ports on that card to be LAN right? Then the extra ports are just behaving like a switch, correct? Why doesn't it work to just include the virtual interface in that bridge?

I think I'm getting what you're talking about with not passing through. I'm not trying to push the whole of my LAN through virtually. I'm only talking about providing LAN connectivity to my VM's and Proxmox GUI through the virtual interface. 

Hmm, yeah, it sounds like it has created a bridging loop, which amounts to a broadcast storm, and that is the behavior you saw with it shutting down your network, it sounds like. It may not work since you are combining the two bridges for the one VM if I am thinking about this correctly and that is what causes a bridge loop if there are more than one layer 2 path between devices. So, it may not work to try to include a virtual interface from the Linux bridge.

I guess next time I feel like breaking things again I should try it again and see if I'm able to access my other services and if it's just the Proxmox webGUI that I can't reach. Because if I created a loop, that means the bridge was doing something and unplugging the physical cable should break the loop. 

@life-from-scratch it would be interesting to see what happens with more testing and see if you are still unable to get to the web UI on the same setup or if this was due to something else at the time. Keep me posted on what you find there 👍 

Well.... I unplugged the cable from my switch back to the Proxmox node and inside PfSense just bridged the LAN and virtual interface together and for some reason this time it just seems to work. All my services seem to be up and I can access the Proxmox dashboard. So I'm not sure why it didn't work when I initially set it up, but it does now.

This opens up a lot more options for machines to run a "forbidden router" setup on like Hardware Haven did. I get why bare metal routers are preferred, but I also like the idea of being able to run other critical services (Home Assistant for me) on the same machine. Running like this it should work with only two nics. One for WAN, one for LAN and not needing a third to for the hypervisor. 

@life-from-scratch that is great! That is cool that it works as you expected it to the first time. Keep me posted on how it performs and operates moving forward.