hyperkube proxy, kubelet can't find iptables chain, rkt run --net=host
After much fault isolation, I've found the cause and solution.
In my case, I'm running a custom kernel pkg (linux-image), which was missing several kernel modules related to iptables. So when kubelet tried to append iptables rules that contained a comment, it errored because xt_comment wasn't loaded.
These are the modules I was missing: ipt_REJECT, nf_conntrack_netlink, nf_reject_ipv4, sch_fq_codel (maybe not required), xt_comment, xt_mark, xt_recent, xt_statistic
To get a complete list of modules that I likely needed, I logged into a CoreOS kubernetes worker and looked at its lsmod. Then just compared that list to my "problem" machine.
I had this issue on a gentoo box with a custom kernel configuration whilst running k8s using rancher's k3d 1.3.1. Rebuilding the kernel with all the sane iptables + xt_comment solved this issue for me.