Block outgoing connections to private IPs from Docker containers

You are adding the rules in the wrong chain. The traffic that is originated from a docker container passes through the FORWARD chain of the filter table, not the OUTPUT chain. This is because from the host computer's perspective, the traffic is incoming from the docker0 interface, and the host computer is merely acting as a forwarder.

In order to differentiate between inbound and outbound traffic, use the -i and -o options to specify interface. Also you can't use uid to determine whether the traffic is coming from a docker container (since the data is not locally originated). Checking incoming interface is enough for that.

So, add the following rules to the DOCKER-ISOLATION chain (which is being called from the FORWARD chain):

-A DOCKER-ISOLATION -d 192.168.0.0/16 -i docker0 ! -o docker0 -j REJECT --reject-with icmp-port-unreachable-A DOCKER-ISOLATION -d 100.64.0.0/10 -i docker0 ! -o docker0 -j REJECT --reject-with icmp-port-unreachable-A DOCKER-ISOLATION -d 172.16.0.0/12 -i docker0 ! -o docker0 -j REJECT --reject-with icmp-port-unreachable-A DOCKER-ISOLATION -d 10.0.0.0/8 -i docker0 ! -o docker0 -j REJECT --reject-with icmp-port-unreachable

Replace docker0 by name of the virtual interface created by docker.

(Note: If the chain DOCKER-ISOLATION doesn't exist, append directly to FORWARD chain).

Also look at the output of iptables -vL and iptables -t nat -vL to better understand how addresses are being translated.