Kubernetes networking issue - Service nodePort can't be reached externally
I found the cause of why the service couldn't be reached externally. It was because iptables FORWARD chain was dropping the packets. I raised an issue with kubernetes at https://github.com/kubernetes/kubernetes/issues/39658 with a bunch more detail there. A (poor) workaround is to change the default FORWARD policy to ACCEPT.
Update 1/10
I raised an issue with Canal, https://github.com/projectcalico/canal/issues/31, as it appears to be a Canal specific issue. Traffic getting forwarded to flannel.1 interface is getting dropped. A better fix than changing default FORWARD policy to ACCEPT is to just add a rule for flannel.1 interface. sudo iptables -A FORWARD -o flannel.1 -j ACCEPT.