kubernetes-1.6: build is never ends

[Bregor]

Not sure which repo is appropriate for this issue, so will leave it here ;)

Environment:

$ kubectl version --short 
Client Version: v1.6.1
Server Version: v1.6.1

$ helm list
NAME                    REVISION        UPDATED                         STATUS          CHART                           NAMESPACE
deis                    2               Fri Apr  7 12:00:58 2017        DEPLOYED        workflow-v2.13.0                deis

DC:

• GKE
• Bare Metal

Logs:

$ stern -t -n deis 'builder|controller'

deis-builder 2017-04-07T14:04:31.785381665Z Accepted connection.
deis-builder 2017-04-07T14:04:32.948655499Z Starting ssh authentication
deis-controller 2017-04-07T14:04:33.011059142Z 10.104.4.62 "GET /v2/hooks/key/f5:99:9d:d8:61:44:d4:7b:df:e2:34:a1:c7:c9:7e:26 HTTP/1.1" 200 88 "deis-builder"
deis-builder 2017-04-07T14:04:33.361007425Z Channel type: session
deis-builder 2017-04-07T14:04:33.361058635Z
deis-builder 2017-04-07T14:04:33.529354005Z Received request of type auth-agent-req@openssh.com
deis-builder 2017-04-07T14:04:33.529402743Z
deis-builder 2017-04-07T14:04:33.529466321Z Key='LANG', Value='ru_RU.UTF-8'
deis-builder 2017-04-07T14:04:33.529517263Z
deis-builder 2017-04-07T14:04:33.529884481Z receiving git repo name: portby-staging-test.git, operation: git-receive-pack, fingerprint: f5:99:9d:d8:61:44:d4:7b:df:e2:3:a1:c7:c9:7e:26, user: vkfont
deis-builder 2017-04-07T14:04:33.529921485Z creating repo directory /home/git/portby-staging-test.git
deis-builder 2017-04-07T14:04:33.569422888Z writing pre-receive hook under /home/git/portby-staging-test.git
deis-builder 2017-04-07T14:04:33.5697572Z git-shell -c git-receive-pack 'portby-staging-test.git'
deis-builder 2017-04-07T14:04:35.678336614Z Waiting for git-receive to run.
deis-builder 2017-04-07T14:04:35.678382852Z Waiting for deploy.
deis-controller 2017-04-07T14:04:36.22369714Z 10.104.4.62 "POST /v2/hooks/config/ HTTP/1.1" 200 526 "deis-builder"


$ git push deis master

-----> Build succeeded!
-----> Discovering process types
       Default process types for Node.js -> web
-----> Checking for changes inside the cache directory...
       Files inside cache folder changed, uploading new cache...
       Done: Uploaded cache (15M)
-----> Compiled slug size is 33M

Nothing happens after it.
Timeout indicates with

---> ---> ---> ---> ---> ---> ---> ---> ---> [ERROR] Failed git receive: portby-staging lock exceeded timeout

after five minutes of waiting.

As possible solution I tried to add exclusive rights for all deis-related ServiceAccounts with following clusterrolebinding without any success:

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: deis:destroy-all-the-things
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: default
  namespace: deis
- kind: ServiceAccount
  name: deis-builder
  namespace: deis
- kind: ServiceAccount
  name: deis-controller
  namespace: deis
- kind: ServiceAccount
  name: deis-database
  namespace: deis
- kind: ServiceAccount
  name: deis-logger
  namespace: deis
- kind: ServiceAccount
  name: deis-logger-fluentd
  namespace: deis
- kind: ServiceAccount
  name: deis-monitor-telegraf
  namespace: deis
- kind: ServiceAccount
  name: deis-nsqd
  namespace: deis
- kind: ServiceAccount
  name: deis-registry
  namespace: deis
- kind: ServiceAccount
  name: deis-workflow-manager
  namespace: deis

[afoninsky]

Same issue here. GKE 1.6.0

[bacongobbler]

Additional context from slack, which on first impression appears to be a kubernetes issue:

deis      2017-04-07 12:11:36 +0300 MSK   2017-04-07 12:11:36 +0300 MSK   1         slugbuild-portby-staging-ba3e805b-411a9f50   Pod                 Warning   FailedMount   kubelet, gke-usa-production-himem-f4630381-06z4   Unable to mount volumes for pod "slugbuild-portby-staging-ba3e805b-411a9f50_deis(d873a412-1b71-11e7-9a69-42010a800009)": timeout expired waiting for volumes to attach/mount for pod "deis"/"slugbuild-portby-staging-ba3e805b-411a9f50". list of unattached/unmounted volumes=[objectstorage-keyfile portby-staging-build-env default-token-60nnj]

@Bregor can you check and see if those secrets exist? Not sure what default-token-60nnj is

[Bregor]

@bacongobbler this message appears only after deploy timed out (if it matters).

[Bregor]

@bacongobbler default-token-60nnj is present.

[Bregor]

BTW, is there any intersections with this issue: deis/workflow#372?

[bacongobbler]

It's possible, yes. Both @mboersma and myself haven't had issues pushing apps when testing v2.13.0 on Azure or Minikube on k8s v1.6.1.

[jordan-thoms]

Any updates on this issue? It's preventing me from trying out deis, as our cluster is on 1.6 and due to separate bugs the downgrade of kubernetes from 1.6 to 1.5.6 is also broken.

[bacongobbler]

If it's something on the kubernetes side of things, there is nothing we can do other than supply an issue to GKE. Have you tried on other cloud platforms?

[Bregor]

Same here on bare metal.
BTW, @bacongobbler I can make empty bare metal setup with 1.6 and workflow and provide root access for you, if you want to touch it personally.

[mboersma]

Just confirming: I was eventually able to reproduce this issue on Kubernetes 1.6.0 on GKE and on minikube with both Workflow v2.12.0 and v2.13.0. Doesn't happen on Kubernetes v1.5.3.