Hello beautiful people of this community!

I asked for your help a month back about setting up OPNsense on Proxmox. I went through all the comments in the post.

I got a bit distracted by IRL commitments but I sat down with it properly last week. After a lot of studying, and a bit of trial and error, I have done it!

My Journey

This was my first time with Proxmox, so I spent a few weeks with Proxmox initially before undertaking OPNsense. OPNsense took me a week to install and deploy. I did a staggered deployment over multiple days, i.e., installed the VM first, check connectivity, check DHCP, and then make it part of my homelab.

Architecture

The diagram might look complicated, but it’s simple.

My host has 3 interfaces, onboard NIC and a dual NIC card. I made a LAN and WAN interface out of the dual NIC card, and made the onboard NIC as a management interface. I connected all of the interfaces to dedicated Linux bridges and the underlying OS/VMs only use the Linux bridges. The onboard NIC is only used to configure Proxmox in case something catastrophic happens. The other VMs can communicate with the OPNsense VM via the virtual LAN interface.

I initially was thinking of going through the pass-through option, but after some research, I saw the upsides were very low and you lose a bit of flexibility.

Challenges

  1. KEEP TRACK OF YOUR NIC TO LINUX BRIDGE MAPPING - This caused me a lot of stress and trails. I thought I was trying to ping the LAN interface, was connected to the WAN bridge.
  2. The Chicken & Egg Problem - During boot, Proxmox needs an IP but the DHCP service inside OPNsense is not up yet. Without that IP, I cannot log into Proxmox. I solved it using a hard-coded IP for Proxmox.
  3. Options galore - Coming from standard routers, OPNsense is a different beast; it has so many options that it becomes a crazy labrynth. Had to read again and again to understand what the options meant.
  4. Where is dark mode? - The first change that I wanted to do was set dark mode and I don’t know why but it an ungodly amount of time for something so simple. It’s present in such a weird location (System -> Settings -> General -> Theme)
  5. This ain’t Kansas anymore - This is OpenBSD, not Linux. So the same commands won’t work. There is drill, no dig.
  6. PLEASE CLICK APPLY - Many times I just forgot to click Apply after making a change and I got super confused.
  7. DNS, YOU DEVIL - This was the setting which made me question my own sanity. I have a simple but non-standard setup for DNS. I am running Pihole (Ad-blocking + Local domains) + Adguard DNS Proxy (Supports encrypted DNS). I wanted to tell OPNsense “Hey, use this IP to resolve all internal and external DNS queries”. OPNsense was just not having it. It took me two days of experimentation to finally figure out the right approach (Disable WAN populated DNS, turn off Unbound, and use DHCP options).

Thanks to these people

Their comments really helped me out (Technical and encouragement)

Closing remarks

I am still documenting the entire process. I will publish the entire process of how I got a bare Lenovo M720Q working as a virtualized OPNsense router. Remember, if I can do it, anyone else can do it.

  • Shadow@lemmy.caEnglish
    3·
    10 hours ago

    I’d also suggest throwing more cores at it than you think it needs, and increasing the nic buffer sizes. This is what finally solved the random 1-2 minute outages I was seeing on lemmy.ca, turns out we were somehow saturating the queues even at relatively low traffic levels. (these issues might be specific to our model nic)

    Finally, make sure hardware offloading is disabled, and your CPU type is set to host.