DragonflyBSD Home Router
My trusty PCEngines APU2 is giving-up-the-ghost. Sadly the AMD SoC it uses is EOL, and PCEngines is thus winding up. So I need to replace it. Fortunately, I have a Beelink SER5 kicking about (a second one). It only has one NIC, but 802dot1q never killed anyone.
Base requirements
For this to work, I need a few things set up on this new router:
- A real interface with an address I can use to get a shell
- A virtual/psuedo-interface tagging frames with a non-default VLAN
- A PPP client talking to my ISP-provided modem over that non-default VLAN
- A router-solicitation daemon to request an IPv6 address/prefix from my ISP
- A DHCP daemon to inform clients of their IPv6 addresses
- A router-advertisement daemon to inform clients of my IPv6 prefix
- A firewall to perform NAT and to protect the edge of the network
- A recursive DNS server to resolve lookups and block ads
- Some monitoring to ensure my router is healthy
In addition, I need my switch to have at least 8x1000BASE-T ports, and support for configuring VLANs. What I ended up buying well overshot that - the Extreme Networks X440-G2-12T-10GE4. Silent unless running unusually hot (the fans only kick in past a threshold), it's a nice edge switch with 4x10GBASE-X uplinks and 12x1000BASE-T switchports.
I bought it from etb-tech, who I've used professionally in the past. They're alright; quite good at sorting out problems that pop up (dead PSUs, misconfigured units, and so on).
Extreme X440-G2-12T-10GE4
I've never used Extreme Networks kit. Some Cisco, a lot of Dell, but not Extreme. But - it's purple, shouldn't have fans spinning during normal operation - and it seems to be a bit more than what I need. So I figured I'd give it a go.
Intial configuration was done over a serial cable ("rollover", RJ45 to DB9). ETB were good enough to properly reset things before shipping it out. I wont cover the commands I ran (alas, I didn't jot them down). To summarise:
- Get through the initial setup wizard
- Assig an IP address to the default VLAN
- Enable SSH
- Set switchport 1 to PVID 10 (assign s1 to VLAN 10 'untagged')
- Set switchport 2 to PVID 1 (the default), and assign to VLAN 10 ('tagged')
- Set up an snmpv3 user for monitoring
DragonflyBSD
rc.conf
Aside from typical host configuration that is generally populated as part of the installation process, a few settings need changing in order to prepare the box to function as a router:
gateway_enable="YES" ipv6_enable="YES" ipv6_network_interfaces="re0 tun0" ipv6_gateway_enable="YES" ipv6_router_enable="YES"
VLANs
VLANs aren't much of a bother on DragonflyBSD. I don't know if it supports (for example) 802.1ad (q-in-q), but it certainly does support typical use. The syntax to create a named VLAN interface in rc.conf is interesting:
vlans_re0="vlan10" create_args_vlan10="vlan 10"
While this works (and is suggested by rc.conf), the same thing can be done a different way (as found in examples for FreeBSD online). This alternative has the benefit of also creating the tun0 interface, which I'll need later.
cloned_interfaces="vlan10 tun0" ifconfig_vlan10="vlan 10 vlandev re0"
So - simple enough. This VLAN is used to push tagged frames carrying PPP over the switch (configured above) and onto the modem, so it doesn't need further configuration.
PPPoE
PPP is less simple. For reasons unclear to me, pppd took umbrage at the thought of creating its tun interface, and would instead complain it doesn't exist and exit. Thus the cloned_interfaces change I made above.
I'm not sure how this will interract with wireguard, which also uses a tun interface (I'm presuming when kernel support lands in 6.6, it'll just work). Aside from that, things seemed quite standard. The FreeBSD documentation out there helped a lot. For PPP, in rc.conf:
ppp_enable="YES" ppp_profile="aaisp" ppp_mode="ddial" rtsold_enable="YES" rtsold_flags="-d tun0"
And in ppp.conf:
default: set log Phase Chat LCP IPCP CCP tun command set device PPPoE:vlan10:aaisp set ctsrts off set timeout 0 set redial 0 0 set mtu 1492 set mru 1492 enable ipv6cp enable ipcp enable dns enable mssfixup enable lqr shell /usr/sbin/ndp -i re0 -accept_rtadv shell /usr/sbin/ndp -i tun0 accept_rtadv add! default HISADDR add! default HISADDR6 aaisp: set authname my_username set authkey my_password
You'll note the addition of ndp commands. This is simply to ensure that the local network interface doesn't accept router adverts, while ensuring the ppp interface does. Do I expect spurious router adverts on my local network? No, but it would be an interesting attack to perform on a local network. There's no harm in making sure we're not accepting those packets.
PF
PF is a large topic. The only thing I'll comment on here, is that the version shipping with DragonflyBSD is not the same as the version in OpenBSD, nor FreeBSD. Both in terms of code, and in terms of version. For this reason, you can really only depend on the man-page for pf.conf (well, or old threads on the FreeBSD forums). The best example are the changes to nat and rdr, which are now done with match rules upstream.
IPv4 addresses with dhcpd
dhcpd is a dog. But I've always felt odd about using dnsmasq when I don't actually need any of its DNS functionality. Configuration for my network, specified in dhcpd.conf (which I mixed up with dhcpcd.conf at least once), is as follows:
option domain-name "piconet.co.uk"; option domain-name-servers 192.168.1.254; default-lease-time 600; max-lease-time 7200; authoritative; log-facility local7; subnet 192.168.1.0 netmask 255.255.255.0 { range 192.168.1.20 192.168.1.120; option routers 192.168.1.254; }
IPv6 RA's with radvd
The easiest part - telling radvd what interface to run on is enough. It'll pick up my prefix from the interface's own address allocation. Again in rc.conf:
rtadvd_enable="YES" rtadvd_interfaces="re0"
DNS with Unbound
I'd already been running unbound on a different box, which OPNSense had been providing to clients. So this was a case of copying that configuration over and ensuring everything was running properly. In unbound.conf:
server: interface: re0@53 prefer-ip4: no prefer-ip6: yes access-control: 192.168.1.0/24 allow access-control: 2001:8b0:ca70:32b2:: allow access-control: fe80:: allow cache-max-ttl: 3600 cache-max-negative-ttl: 15 forward-zone: name: "." forward-addr: 2001:8b0::2020 forward-addr: 2001:8b0::2021
Monitoring
So, one thing OPNSense does really well is its dashboard. I mean, yes, php-fpm does sometimes die, and restarting the service doesn't always work. But it's a nice interface that provides status at a glance. I wanted something like that for both my new router and my other DFly-based server.
Munin is one option. But is backed by a PostgreSQL instance, which is a bit much. Netdata is interesting, in that they specifically support FreeBSD in both their open-source and paid/hosted product. But they don't support Dragonfly, and I really don't want to go through the rigmarole of building from source and figuring out how to add support. Icinga is an option. I have considerable experience using it, but again, it requires a dedicated database (Maria or Postgres).
Whereupon I stumbled across Monit. It's in the package repos, it has one configuration file, and it has built-in support for monitoring disks, network interfaces, and remote hosts (via ICMP/TCP/UDP). The only downside is that to monitor multiple hosts from one monit instance, you seem to need to pay for m/monit.
I'm not against paying for software. Especially not when we're talking about free (as in freedom) software. But the initial pricing is too steep for me. One host costs 41EUR, thereafter each additional host costs +9EUR. I only need to monitor two hosts, so that initial cliff is quite large.
Still! The open-source single-host version works fine. It just means I have one interface per host.
Closing Notes
It's been fun setting this up. If not for that one weird issue with PPP not creating tun0 on start, I'd have said it was painless. IPv6, as always, was a bit fiddly. Seeing EXOS and comparing it to whatever Dell ships on their PowerEdge switches was fun (Dell wins on functionality in their HTTP interface, Extreme win on how they present functionality).
Will it last? There's part of me that feels I should probably just slap OPNSense on it and be done with it. But on the other hand, revisiting this stuff is probably helpful for my job. If anything changes, I imagine it'll get a post.