The Problem

While it’s not obvious at first blush, the default configuration for firewalld and docker is… bad. At least on systems that do not use iptables as their back end firewall technology. Like, your firewalld rules don’t apply to traffic destine for docker containers bad. An issue discovered at my work when one of our coworkers asked ‘Hey, why can I get to the new ticket system without vpn?’.

This is due to the fact that while firewalld is managing nftables or other firewall back ends, Docker continues to lean on iptables for it’s filtering and forwarding. This results in the forwards sidestepping firewalld entirely exposing the docker ports to the wild. This is a problem if you work somewhere with hosts sitting on the public internet. In the case of our deployments the older CentOS 7 systems didn’t manifest this behavior because we had them default to iptables instead. However our new systems use Rocky 9, and that revealed this particular weakness.

The Goal

FIX THIS. But more seriously, the goal is to keep our firewalld automation in place and have those configurations honored by docker. To do this we are going to have to make some big and small changes both to docker and firewalld.

Docker Modifications

There’s three major things to change, iptables automation, permanently created docker networks, and your compose files. Updating each of these will have impacts on the docker daemon, or specific running containers. So this will involve a downtime for the component parts as you update them.

IPTables

First up, you need to disable the docker IPTables automation. This can be done in a few ways depending on how you manage docker, for a vanilla install you can simply add {"iptables": false} to your daemon file (/etc/docker/daemon.json). And restart the service with systemctl restart docker. Initially this will not change anything, as demonstrated by iptables -S, however if you run firewall-cmd --reload it will nuke those rules and everything will break (this is fine).

Docker Networks

For docker networks, we need to solve a thorny issue that shows up because of the change above. Unfortunately the iptables flag in the docker daemon is overloaded and pulling double duty to also add virtual network adapters to the docker firewall zone in firewalld. This is required for containers to be able to receive traffic. So, for any networks that you create you must choose a predictable name for that network. This can be done with the --opt command while creating the network. After that you can add that named network to the firewalld zone permanently. For example we use a proxy network for all containers that need to be made available via Traefik. We’ll need to recreate that network with a fixed name as follows (note: if you have a pre-existing named network, you will need to remove it first):

$ docker network create --opt com.docker.network.bridge.name=proxy proxy
$ firewall-cmd --zone=docker --add-interface=proxy --permanent
$ firewall-cmd --reload ## required to enable permanent changes

The above must be done for all custom networks that do not use docker0 as their virtual network interface.

Docker Compose

A similar update to above has to be done for compose files, however this ones a bit more awkward. For each compose entry you wish to use you need to choose a name for each of the related network. We’ll be substituting that in for <vnet_name> in each of the below code samples to make things work.

First lets add the name to the firewalld allows:

$ firewall-cmd --zone=docker --add-interface=<vnet_name> --permanent
$ firewall-cmd --reload  ##required to make this configuraiton active

Then lets update the docker compose file, you’ll want to add the following if you have a ‘free standing’ compose file for each service entry (ie, you aren’t using say a traefik proxy along with this):

services:
  <service_name>:
    networks:
      - <vnet_name>

And then at the bottom of the compose file add:

networks:
  <vnet_name>:
    driver_opts:
      com.docker.network.bridge.name: <vnet_name>

This will force compose to create bridge networks of the specific name and ensure all defined services are joined to that network name.

If you wanted to do this for a system that was using traefik and an external proxy network it would look like this:

services:
  <service_name_that_needs_reverse_proxy>:
    networks:
      - <vnet_name>
      - proxy
## ...      
networks:
  proxy:
    external: true
  <vnet_name>:
    driver_opts:
      com.docker.network.bridge.name: <vnet_name>

Firewalld Modifications

This comes down to two quick things. First any firewalld zones you wish for docker containers to be able to transmit out to via nat, you will need to add the masquerade command. And as mentioned above, you will need to make sure all docker interfaces are in the docker zone so forwarding works.

Masquerade Interfaces

If you wanted to enable masquerade on your default zone, which for many configurations is the only thing you will need, the following will get you setup:

$ firewall-cmd --add-masquerade --permanent
$ firewall-cmd --reload ## required to activate permanent rules

At this point, all docker containers should be able to go out your default interfaces. If you have any other active zones, for some multi-homed devices for example, you will want to add --zone commands above for those additional active zones. You can see active zones with firewall-cmd --get-active-zones.

Zoning Docker Interfaces

There’s a quick additional tweak that needs to be made here. You simply need to add the docker0 interface into the docker zone so it’s there by default. The following will add this for you:

$ firewall-cmd --zone=docker --add-interface=docker0 --permanent
$ firewall-cmd --reload ## again required to setup permanent changes

Conclusion

All in all this isn’t that annoying, but the fact that it breaks the ‘just works’ nature of docker is a huge bummer. I wish that wasn’t the case if I’m being totally honest. At work we are migrating all of this to be managed by our puppet layer so I’ll be adding a puppet fact that detects the docker network names and adds them to the run time docker zone, and purges them when they are no longer in use to replicate this behavior, but honestly. That shouldn’t be necessary, but in the mean time I hope you find this helpful!

Areas of Improvement

Unfortunately the ‘proper’ fix for this is in the court of Docker / the Moby project. If they added a firewalld flag along side the iptables flag that simply added the virtual network device names to the docker zone and enabled masquerade for publicly facing zones.

It should also be noted, all of the above only applies to IPV4. For IPV6? I have no idea, we don’t currently use it at work and I’ve got no use for it at home. I wouldn’t be surprised if it had some weird behavior there as well tho.

Post Publication Gotchas

12-7-23

We found at work that despite best efforts to avoid it, no matter what we did we could not get firewalld to fully dispose of the iptables rules without a full reboot on one of our servers (rock9). After a reboot it worked exactly as expected.