init container fails with iptables: Chain already exists

[debianmaster]

init container fails with iptables: Chain already exists. message on openshift, any clue?

oc logs podname -c init gives me iptables: Chain already exists.

Version information
OpenShift Master: v3.5.5.5
Kubernetes Master: v1.5.2+43a9be4

[debianmaster]

i have following patch on deploymentConfig

[{"name":"init","image":"docker.io/istio/init:0.1","args":["-p","15001","-u","1337"],"resources":{},"imagePullPolicy":"Always","securityContext":{"capabilities":{"add":["NET_ADMIN"]}}},{"name":"enable-core-dump","image":"alpine","command":["/bin/sh"],"args":["-c","sysctl -w kernel.core_pattern=/tmp/core.%e.%p.%t \u0026\u0026 ulimit -c unlimited"],"resources":{},"imagePullPolicy":"Always","securityContext":{"privileged":true}}];

[debianmaster]

This appears to be an issue with enabling SELINUX, i had to disable selinux on all nodes to make it work.

[ldemailly]

With netadmin request it should work, or requesting specific access to iptables rather than having to disable selinux entirely?

Also why is it saying chain already exist, should be permission denied?

[bjartek]

We have the same issues trying to get it to work. For us disabling selinux is not an option. Any more information on how we can solve this will be very helpfull.

oc v3.5.5.8
kubernetes v1.5.2+43a9be4
features: Basic-Auth GSSAPI Kerberos SPNEGO

istio 0.1.6

I followed debianmasters guide in order to try and set it up.

[debianmaster]

None as of this release. Afaik

[ayj]

The init-container needs sufficient permissions to write iptable rules for redirecting inbound/outbound traffic to the sidecar proxy. Injection requires CAP_NET_ADMIN though there seems to be potential issues with RBAC (see here) which were resolved by requesting broader privileges, i.e. "securityContext":{"privileged":true}.

With regards to SELINUX specifically, you could try some of the steps described here. It looks like you'll need some combination of updating the selinux policy and/or adding the necessary selinuxOptions to the security context of the init-container. @debianmaster, is there anyone on your side who could help with recommendations here? Once the recipe is worked out we can look into ways to rolling that support back into Istio proper alongside the existing mechanism(s).

[bjartek]

I got the init container to run with SELinux enabled. I modifed the injected init cointaner to add priviledged=true to the first init container.

"securityContext":{"capabilities":{"add":["NET_ADMIN"]},"privileged":true}

Your comment with mentioning requesting broader priviledges what was got my thinking of it @ayj because only the second init container was started with priviledged=true, not the first one.

If you can point me at the code I can always contribue the fix to the injected init containers. Or it might be easier for somebody with commit priviledges already to do it.

[bjartek]

It does not look like the iptables rules from https://github.com/istio/pilot/blob/master/docker/prepare_proxy.sh are present in my iptables on my nodes. I can see lots of rules from the kubelet but not this line:
iptables -t nat -N ISTIO_REDIRECT -m comment --comment "istio/redirect-common-chain"

I have done GET request towards my services but nothing shows up in grafana or servicegraph. Not so strange if the iptables rules are not present.

[rshriram]

Try to exec into your container in the pod and curl httpbin.org. See if that works. If the iptables rules are installed, it shouldn't.

On Fri, Jul 7, 2017 at 3:32 AM Bjarte S. Karlsen ***@***.***> wrote: However even though everything now is running I am not sure that the init container actually manges to modify iptables. What is it supposed to set? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#34 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AH0qd3Z07z6XJfBumsfma6F2cPotWkw3ks5sLd8ggaJpZM4N7yHc> .

-- ~shriram

[ayj]

It does not look like the iptables rules from https://github.com/istio/pilot/blob/master/docker/prepare_proxy.sh are present in my iptables on my nodes.

Our init-container programs iptable rules in the pod's network namespace. The kubelet (kubeproxy) writes iptables in the node's network namespace. You'll need to exec into the pod with privileges (or equivalent via docker commands) to dump iptable rules written by prepare-proxy.sh.

@rshriram's negative test suggestion would also work.

[ayj]

If you can point me at the code I can always contribue the fix to the injected init containers. Or it might be easier for somebody with commit priviledges already to do it.

PR are always welcome and I can help with review/merges. Looks like you already found prepare_proxy.sh which is the correct place to make these kinds of fixes for now.

[bjartek]

I got it to work. I had tampered with the istio-mixer service since prometheus reported it could not scrape the 42422 port. But it looks like it works when there has been traffic.

I will send a PR to add the priviledged=true port to the injected init container in a while.