No network access in Docker containers on Arch Linux

I recently rented a dedicated server from Hetzner, upon which I installed an encrypted Arch Linux system, and needed to migrate over some services. For isolation and easy dependency management I make heavy use of containers. Installation of docker and docker-compose was simple enough, but when starting the first service container it became clear that it lacked network access.

Last time I experienced similar issues the cause was UFW (both UFW and docker works by manipulating iptables – a rabbit hole worth diving into if you’re interested). This time however I went with an external firewall.

Considering it was a fresh system and I hadn’t made any modifications to the local firewall, I wasn’t expecting any misbehaving rules. Nevertheless I took a glance at the rulesets with iptables -nL and found nothing.

As the system was installed using an installation image provided by Hetzner, my next thought was that perhaps the installation came with some customized settings which could explain the behaviour. Performing a search through /etc I indeed found some customizations. One stood out:

Configuration files under /etc/sysctl.d contains parameters instructing the kernel on features it should enable and/or disable.

A part of the magic of docker is how it sets up and manages the container networking for you. I won’t go into detail on how it works under the hood but it’s worth a read. The marked line above tells the kernel about whether or not it should allow IP forwarding, something dockerd needs to work it’s magic. In this case it was commented out. It wasn’t an issue however as sysctl -a | grep forward showed that IP forwarding was enabled already. Explicit is better than implicit, and to avoid future issues I uncommented the line.

Note: It didn’t occur to me while troubleshooting but I’m uncertain on whether the iptables FORWARD chain would have shown up at all in the listing if IP forwarding wasn’t enabled.

Sifting through the system logs nothing stood out apart from the system being installed with the public ethernet interface “misnamed”:

Unsure on how to proceed I took to search the Internet for answers and stumbled upon this thread which looked similar. According to the linked Arch wiki entry there could be a conflict happening – dockerd tryng to enable IP forwarding while systemd-networkd is disabling it. In effect causing a stale mate. I ran networkctl list to verify, and indeed, it showed the docker bridge, docker0, being stuck in the configuring state.

As a remedy I created 20-docker.network and 21-veth.network under /etc/systemd/network, explicitly telling systemd-networkd to not interfere with docker. I also added the recommended IPForward=yes to the physical ethernet interface configuration:

After restarting the network by running systemctl restart systemd-networkd, it worked! 🥳