Welp, it happened. First, some context: I run a PfSense firewall between my WAN connection and my homelab. You can see it in this post where I made a custom power switch for it and in this post where I connected the power switch up to some other switches. For the storage of the firewall I originally used an 8 GB USB stick, because it saved space and was nice and small. I read online that it's possible that the USB stick dies because of the very frequent reads and writes that a hard drive puts up with. However, I though "Ahh, what's the worst that could happen?" Well, the USB stick died and the firewall couldn't boot anymore 😓. The file system had become corrupt and couldn't be booted anymore. I was greeted with this message when powering on the firewall:
I already knew I had to reinstall the OS of the firewall, but I didn't want to have to configure the firewall. I read online that PfSense saves it's config to a config.xml file, so all I needed to do was to find it. I first tried to mount the USB stick in on a Ubuntu machine. Because it was a ZFS drive, I had to first figure out how to mount this drive. Unfortunately, the filesystem on the drive had gotten corrupt 😓. I tried some other options, but in the end, I found out that recovering on Ubuntu was not an option. Then, I searched for the best way to mount and recover ZFS drives, and found out that FreeBSD was a good OS to do this. To get started, I burned the installation file to a disc:
After FreeBSD booted, I mounted the drive and tried to look through it's contents. However, I got the message R/W Mount of /mnt/usb Denied. Filesystem is not clean - Run fsck. According to this error message, I had to use fsck. When running this on the drive, It asked me to fix certian parts of the drive and I just said yes to all of them. After that, I could mount the drive successfully, but it seemed fsck had only recovered some memory blocks, as the drive contained only folders that had memory addresses as names (#00000, #08000, #15200 etc.). Some of these contained actual files, so I just started looking through them. And alas, I found the config file! It was apparently located at memory block #16052 and the whole file was intact!😃 I mounted another drive and copied the file to it. Now, I only had to reinstall PfSense and import the config.xml file. I also switched out the USB stick to a 500GB 3.5" hard drive that I had laying around, to avoid it crashing again. To reinstall PfSense, I made an install disc:
After that I imported the config.xml file during the install and the firewall was back up!
Don't try to cut corners and think "Ah, that won't happen to me". It will. And you'll spend your evening trying to recover files from some cheap USB stick 😬