DragonflyBSD and NVMM and HAMMER2

DragonflyBSD and NVMM and HAMMER2

In addition to rewriting the site, I've finally gotten around to migrating everything back to DragonflyBSD.

Moving away from Dragonfly

I don't recall why I stopped using DragonflyBSD. I think some of it came down to wanting to use containers for the self-hosted services I run. Nevertheless, were that not something I wanted at the time, Dragonfly doesn't easily run software like Misskey (very modern NodeJS) or Jellyfin (.net core - people apparently use this to make free software?). So the switch may have been imminent anyway.

Misskey could, of course, be replaced with something like Pleroma. Pleroma does run on DragonflyBSD without issue. But I've already taken the time to write a client class for Ruby (for all it only covers the small scope of Misskey I require). So I'd rather stick with it in some way.

Jellyfin is a bit of a mess. There's no .net core runtime for FreeBSD, let alone DragonflyBSD. Frankly, it's also a bit buggy. I only need it to stream music to an iPhone, ideally with a navigation structure of /Genre/Album/Track. I've always found the process of indexing to be troublesome - missing art (it's just folder.jpg, come on), missing genre, missing albums. The metadata will be in the database, in an NFO, and certainly visible with ffprobe. But nothing.

Nevertheless, Finamp is a phenomenal mobile client. I can't recommend it enough. It is the only reason I persist with Jellyfin over using less messy solutions like Navidrome.

Moving back to Dragonfly

I have a nice mini-PC from a company named "Beelink" I've been wanting to migrate my home server over to, and thought to give Dragonfly another go. I never actually stopped using Dragonfly, but this was in a capacity I hadn't touched in a while. Naturally, I tried and failed to get both Misskey and Jellyfin running. Pleroma and Navidrome did work, but as above; not ideal.

What I ended up doing sounds over-engineered, because it is. But it does work.

NVMM

NVMM is a hypervisor by Maxime Villard. The thing that Qemu talks to for hardware acceleration. Originally written for NetBSD, ported to Dragonfly in 2021. What a champ.

The howto documentation on the DragonflyBSD site is well written and covers everything needed to go from zero to production. It took no time at all to have a running Debian virtual machine I could SSH into, and took almost no time at all to rig it up with an rc.d script (rig is generous) that starts on boot. Thus far I've noticed no CPU contention; it all seems very lightweight. I'd certainly recommend NVMM as a makeshift solution for compatibility to other DragonflyBSD users.

For posterity; a chunk of my rc.conf and my rc.d/misskey service can be found below:

# rc.conf
cloned_interfaces="bridge0 tap666 tap667"
ifconfig_bridge0="up addm re0"
ifconfig_tap666="up"
ifconfig_tap667="up"

misskey_tap="tap666"
misskey_enable="YES"
...

#!/bin/sh

# PROVIDE: misskey
# REQUIRE: LOGIN cleanvar
# KEYWORD: shutdown

# Add the following lines to /etc/rc.conf to enable nginx:
# misskey_enable (bool):        Set to "NO" by default.
#                               Set it to "YES" to enable misskey
# misskey_tap (str):            TAP interface to attach to bridge0
#                               on start.

. /etc/rc.subr

name="misskey"
load_rc_config $name

rcvar=misskey_enable
pidfile="/var/run/misskey-qemu.pid"
sockfile="/var/run/misskey-qemu.sock"

machine="-machine type=q35,accel=nvmm"
cpu="-smp cpus=1"
mem="-m 2G"
net="-netdev tap,ifname=$misskey_tap,id=net0,script=no,downscript=no -device virtio-net-pci,netdev=net0,mac=52:54:00:34:66:60"
rng="-object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0"
disk="-drive file=/usr/nvmm/qcow/$name.qcow2"
vnc="-display none -vnc 0.0.0.0:0"
mon="-monitor unix:$sockfile,server,nowait"
pid="-pidfile $pidfile"

start_precmd="setup_tap"
setup_tap() {
        ifconfig bridge0 addm $misskey_tap
        echo "Added $misskey_tap to bridge0"
}

stop_cmd="stop_vm"
stop_vm() {
        echo "Sending system_powerdown to qemu"
        echo "system_powerdown" | /usr/local/bin/socat - unix-connect:$sockfile 
        sleep 10
}


command="/usr/local/bin/qemu-system-x86_64"
command_args="$machine $cpu $mem $net $rng $disk $vnc $mon $pid -daemonize"

run_rc_command "$1"

HAMMER2 Failure

Attachmed to my server is a two-bay USB3.1 HDD caddy. Great little thing that'll do RAID in addition to JBOD, but I just have it set to present the disks to the OS. The process for preparing these disks, in my mind, was something like this:

gpt init -f /dev/da8
disklabel -e /dev/da8s1
newfs_hammer2 -L H2A /dev/da8s1h
mount @H2A /mnt/H2A

gpt init -f /dev/da9
disklabel -e /dev/da9s1
newfs_hammer2 -L none /dev/da9s1h
mount /dev/da9s1h@LOCAL /mnt/local
cd /mnt/local
hammer2 -t SLAVE -u [clid of H2A] pfs-create H2B
mount @H2B /mnt/H2B

This would, I thought, give me a first disk with a single hammer2 partition acting as a master in a cluster, mounted at /mnt/H2A readwrite. And it did! I also expected it to give me another hammer2 partition, on the second disk, acting as a slave to that master, mounted at /mnt/H2B readonly. It did not! I caused the machine to dump its guts.

I tried it a few ways. Each time I'd end up in a panic. SOFT_MASTER, SOFT_SLAVE, dismounting H2A before mounting H2B, starting fresh each time. One thing I found constantly missing from resourced I'd found on Hammer2 - how on earth do you upgrade a SLAVE to a MASTER. How on earth do you downgrade your MASTER to a SLAVE? The documentation is clear there's an ordered process. But there's no hammer2 pfs-upgrade command.

I relented. I now have two hammer2 partitions, both MASTER (differing pfs clid), with replication handled by cpdup in the crontab. There are worse solutions in the world, I suppose. I do wish I had figured out what I was doing wrong, though. Perhaps it has something to do with the hardware I'm using? Two disks over one USB port? I'd like to replace these disks with SSDs one day, that may be the time to try again.