Complete OpnSense Firewall Migration including HAProxy and Acme with Let’s Encrypt: A Detailed Guide for Self-Hosting Enthusiasts

Migrating a firewall appliance is no small feat, especially when you’ve spent years fine-tuning your configuration. This is the story of my recent journey in upgrading OpnSense, and migrating my HAProxy and Let’s Encrypt settings. Along the way, I encountered challenges, learned valuable lessons, and ultimately achieved success. In this blog post, I’ll walk you through the process and share my insights.

The Upgrade Journey Begins

I embarked on this adventure around 7:30 PM, fully aware of the importance of meticulous planning and backups. The first step was to create backups of the virtual machine (VM) hosting OpnSense. I used a bash script I had previously written to automate this task. This script not only took VM backups but also ensured I had an XML configuration backup of OpnSense. However, a lesson I learned early on was that the “download all” option in OpnSense doesn’t actually download the entire configuration. I had to return later to retrieve the HA Proxy configuration, a crucial component.

By 8:00 PM, I had completed my last backup of the old version, marked as 200238. With backups in hand, I proceeded to create a new VM in the Virtual Machine Manager UI. The process involved unassigning the NIC network cards from the old VM and reassigning them to the new one. This step required careful attention to detail, as I needed to ensure the new NICs matched the old configuration, including PCI slot assignments. Unfortunately, they didn’t match entirely, which caused issues later on.

A Bumpy Start and Quick Adjustments

Once the new VM was up and running, I logged into the console. The next crucial step was to assign the NICs to LAN and WAN interfaces correctly. Due to the earlier mismatch in PCI slot assignments, I initially got the order wrong. To avoid external access to my new OpnSense while I sorted this out, I disconnected the physical WAN cable (I had already changed the root password at this point). After a quick interface assignment reversal and the correct LAN static IP configuration, I enabled DHCP and gained access to the admin console.

Advertisement Begins

Advertisement End

The OpnSense GUI Wizard guided me through an initial setup process, which I completed without much hassle. At this point, I performed another “brute force” backup using my VM backup script (I highly recommend you check out the linked post), creating a backup titled 205130.

Starting from scratch with a blank OpnSense instance was not an option, considering the years of fine-tuning I had spent to allow me to host dozens of production services — everything from a home automation platform, my WordPress blog, and various database systems, to a virtualised firewall appliance, as well as web and mobile analytics tools. — This led to subtle configuration issues over time, resulting in frequent internet dropouts and DNS problems. I also faced challenges with Unbound, which threw errors whenever I attempted to save new settings.

Selective Configuration Restoration

To address these issues, I used the Configuration restore function to selectively restore different aspects of my OpnSense instance. After each restoration attempt, I validated its success. I began with the firewall configuration, followed by NAT, SSL Certificate Authorities, SSL certificates, and the DHCP server.

By 9:00 PM, my home network and internet access were restored, but my blog and external-facing services remained offline due to a lack of reverse proxy service configuration. I created another backup, labeled 210300, at this point.

The next items on the restoration checklist were the Syslog configuration, which allowed me to access logs on the host server itself, and overall System settings. However, while proceeding, the web interface suddenly became unavailable without an obvious reason.

Advertisement Begins

Advertisement End

A Moment of Panic (not really)

Checking the interface assignments revealed no apparent issues, but the web interface remained inaccessible. I knew that the “System” section of OpnSense includes many system-wide settings. It was no surprise that some major had gone wrong.

To troubleshoot this further, I logged into the OpnSense console and used the built-in restore functionality to revert to a prior version of the OpnSense configuration. This was primarily to determine if the issue lay with the System settings restoration.

I referred to my handwritten notes to find out when I had taken the last backup, and correlated this to the closest OpnSense config snapshot.

As it turned out, after a restart, the OpnSense admin console became accessible again. It dawned on me in that moment that I had been accessing the admin console via HTTPS, and the system config restore must have reset the Web UI settings to make the it accessible via HTTP instead. A small detail, but it explains the issue. I applied the System settings again from my OpnSense XML export, changed the Admin URL to HTTP in my browser, and regained access to the admin UI.

Updates, Upgrades, and Challenges

With the basics in place, it was time to install pending system updates and upgrade OpnSense from version 23.7.2 to 23.7.4. Following this, I installed the last remaining plugins: HAProxy and ACME for Let’s Encrypt certificates. Success led to another backup at 9:50 PM, timestamped 215257.

Mounting a Qcow2 Image – or not

The next challenge arose during the import of the HAProxy configuration. I intended to moung the VM disk image and access the configuration files directly on the host machine itself, incorporating thousands of lines of custom HAProxy settings.

For nearly an hour, from 9:50 PM to 11:00 PM, I attempted to mount the virtual machine’s qcow2 image to facilitate my configuration edits following many forum discussions to g. However, I encountered numerous errors along the way. The closest I got was an error message indicating that the disk image itself was write-protected, likely a safeguard implemented by OpnSense to ensure file integrity.

Kitchen Multi-Timer Pro

Now you’re cooking

Multi Timer Pro is your ultimate meal prep companion, keeping track of multiple cooking times and making adjustments on the fly. Give it a try today and become a better home cook!

The SSH Solution

With mounting qcow2 image proving elusive and considering the time I had already spent on this activty, I turned to editing the configuration file manually via the console. Using the Virtual Machine Manager console was not feasible due to its small screen buffer and lack of scrolling capabilities. To work around this, I temporarily enabled SSH on the LAN interface within the OpnSense settings and logged into the instance using Tabby (my SSH client of choice on Windows).

Logging in via SSH using my preferred client, Tabby, I accessed the required /conf/config.xml file using the vi text editor, causing some unexpected nostalgia back to my university days. Fortunately, my memory served me well, and I navigated the editor with utmost precision and ease — though my search history may disagree with that statement. Anyway, one particular command you will find useful is dd and 10dd to delete individual lines of text or any number of lines respectively.

It was during this process that I realized my HAProxy settings export, created using the “download all” function, included only the staging settings, not the production ones. I spent 15 minutes reassigning the PCI NIC cards to my old OpnSense instance to access the admin console and download the correct configuration file.

With the HAPRoxy.conf file in hand, I reversed the NIC assignments, started the new VM, and edited the /usr/local/etc/haproxy.conf file in vi. However, the file comment header advised that OpnSense auto-generates this HAProxy configuration (I have a small gripe with this because it forces you to use the OpnSense UI to configure HA Proxy). Thus, my previous 20 minutes of work were unnecessary, but they served as a valuable reminder.

I jumped to the main config.xml and replaced the entire <HAProxy> tag with my own configuration containing 4536 lines of customisations.

Editing config files like this felt akin to performing brain surgery, with extreme care required in the placement of cursors /(presumably) cutting instruments to avoid breaking the entire system/human while working on one critical aspect.

ACME Certificates and Let’s Encrypt

The next task was importing my current ACME certificates and their corresponding Let’s Encrypt configurations to ensure my production services would be served using HTTPS. Setting this up originally was a time-consuming process, so it was essential to migrate the existing configuration across.

Once again, I accessed the SSH console, opened /conf/config.xml, and deleted the default XML tag. I pasted in my custom <AcmeClient> tag. However, after a restart, I noticed that the import had failed this time, and the ACME section in the admin console remained blank.

Upon inspection, I realized that I had accidentally pasted the <<code>AcmeClient> tag inside the <HaProxy> tag. This mishap was due to a quirky behavior in vi, where the cursor jumped by one line. I corrected the placement, by first reverting to config.xml.backup and reapplying the change. This time, the configuration import was successful.

The Journey’s End

At midnight, I took a final VM backup, capturing the current state of my new OpnSense instance as 001304. The migration and self-hosting adventure had come to an end, and my OpnSense firewall was now upgraded, configured, and secure – serving all my production services once again.

I am also pleased to say that my intermittent connectivity issues and DNS resolver problems with Unbound are now fixed. I suppose the OpnSense upgrade was overdue for some time.

This journey taught me the importance of thorough planning, a methodical approach to system management, and jotting down handwritten notes throughout to serve as reminders should things go wrong and a restore is necessary. Migrating a firewall and setting up self-hosted services can be challenging, but with patience and persistence, it’s achievable. Now, my self-hosted services are running smoothly, and I can rest easy knowing my OpnSense firewall is up to date and secure.

Conclusion

In conclusion, if you’re considering a similar migration and self-hosting endeavor, remember to plan well, take backups, and be prepared for unexpected challenges. With determination and the right tools, you can successfully navigate this journey, just as I did.

Note: This blog post is based on a real-life experience and serves as a guide for those interested in OpnSense / HAProxy and ACME migration.Your specific setup may vary, so always refer to official documentation and seek assistance when needed.