Menu
I was following this documentation to setup Spinnaker on Kubernetes. I ran the scripts as they specified. Then the replication controllers and services are started. But some of PODs are not started
root@nveeru~# kubectl get pods --namespace=spinnaker
NAME READY STATUS RESTARTS AGE
data-redis-master-v000-zsn7e 1/1 Running 0 2h
spin-clouddriver-v000-6yr88 1/1 Running 0 47m
spin-deck-v000-as4v7 1/1 Running 0 2h
spin-echo-v000-g737r 1/1 Running 0 2h
spin-front50-v000-v1g6e 0/1 CrashLoopBackOff 21 2h
spin-gate-v000-9k401 0/1 Running 0 2h
spin-igor-v000-zfc02 1/1 Running 0 2h
spin-orca-v000-umxj1 0/1 CrashLoopBackOff 20 2h
Then I kubectl describe the pods
root@veeru:~# kubectl describe pod spin-orca-v000-umxj1 --namespace=spinnaker
Name: spin-orca-v000-umxj1
Namespace: spinnaker
Node: 172.25.30.21/172.25.30.21
Start Time: Mon, 19 Sep 2016 00:53:00 -0700
Labels: load-balancer-spin-orca=true,replication-controller=spin-orca-v000
Status: Running
IP: 172.16.33.8
Controllers: ReplicationController/spin-orca-v000
Containers:
orca:
Container ID: docker://e6d77e9fd92dc9614328d09a5bfda319dc7883b82f50cc352ff58dec2e933d04
Image: quay.io/spinnaker/orca:latest
Image ID: docker://sha256:2400633b89c1c7aa48e5195c040c669511238af9b55ff92201703895bd67a131
Port: 8083/TCP
QoS Tier:
cpu: BestEffort
memory: BestEffort
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 19 Sep 2016 02:59:09 -0700
Finished: Mon, 19 Sep 2016 02:59:39 -0700
Ready: False
Restart Count: 21
Readiness: http-get http://:8083/env delay=20s timeout=1s period=10s #success=1 #failure=3
Environment Variables:
Conditions:
Type Status
Ready False
Volumes:
spinnaker-config:
Type: Secret (a volume populated by a Secret)
SecretName: spinnaker-config
default-token-6irrl:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-6irrl
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
1h 3m 22 {kubelet 172.25.30.21} spec.containers{orca} Normal Pulling pulling image "quay.io/spinnaker/orca:latest"
1h 3m 22 {kubelet 172.25.30.21} spec.containers{orca} Normal Pulled Successfully pulled image "quay.io/spinnaker/orca:latest"
1h 3m 13 {kubelet 172.25.30.21} spec.containers{orca} Normal Created (events with common reason combined)
1h 3m 13 {kubelet 172.25.30.21} spec.containers{orca} Normal Started (events with common reason combined)
1h 3m 23 {kubelet 172.25.30.21} spec.containers{orca} Warning Unhealthy Readiness probe failed: Get http://172.16.33.8:8083/env: dial tcp 172.16.33.8:8083: connection refused
1h <invalid> 399 {kubelet 172.25.30.21} spec.containers{orca} Warning BackOff Back-off restarting failed docker container
1h <invalid> 373 {kubelet 172.25.30.21} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "orca" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=orca pod=spin-orca-v000-umxj1_spinnaker(ee2511f0-7e3d-11e6-ab16-0022195df673)"
spin-front50-v000-v1g6e
root@veeru:~# kubectl describe pod spin-front50-v000-v1g6e --namespace=spinnaker
Name: spin-front50-v000-v1g6e
Namespace: spinnaker
Node: 172.25.30.21/172.25.30.21
Start Time: Mon, 19 Sep 2016 00:53:00 -0700
Labels: load-balancer-spin-front50=true,replication-controller=spin-front50-v000
Status: Running
IP: 172.16.33.9
Controllers: ReplicationController/spin-front50-v000
Containers:
front50:
Container ID: docker://f5559638e9ea4e30b3455ed9fea2ab1dd52be95f177b4b520a7e5bfbc033fc3b
Image: quay.io/spinnaker/front50:latest
Image ID: docker://sha256:e774808d76b096f45d85c43386c211a0a839c41c8d0dccb3b7ee62d17e977eb4
Port: 8080/TCP
QoS Tier:
memory: BestEffort
cpu: BestEffort
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 19 Sep 2016 03:02:08 -0700
Finished: Mon, 19 Sep 2016 03:02:15 -0700
Ready: False
Restart Count: 23
Readiness: http-get http://:8080/env delay=20s timeout=1s period=10s #success=1 #failure=3
Environment Variables:
Conditions:
Type Status
Ready False
Volumes:
spinnaker-config:
Type: Secret (a volume populated by a Secret)
SecretName: spinnaker-config
creds-config:
Type: Secret (a volume populated by a Secret)
SecretName: creds-config
aws-config:
Type: Secret (a volume populated by a Secret)
SecretName: aws-config
default-token-6irrl:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-6irrl
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
1h 3m 24 {kubelet 172.25.30.21} spec.containers{front50} Normal Pulling pulling image "quay.io/spinnaker/front50:latest"
1h 3m 24 {kubelet 172.25.30.21} spec.containers{front50} Normal Pulled Successfully pulled image "quay.io/spinnaker/front50:latest"
1h 3m 15 {kubelet 172.25.30.21} spec.containers{front50} Normal Created (events with common reason combined)
1h 3m 15 {kubelet 172.25.30.21} spec.containers{front50} Normal Started (events with common reason combined)
1h <invalid> 443 {kubelet 172.25.30.21} spec.containers{front50} Warning BackOff Back-off restarting failed docker container
1h <invalid> 417 {kubelet 172.25.30.21} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "front50" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=front50 pod=spin-front50-v000-v1g6e_spinnaker(edf85f41-7e3d-11e6-ab16-0022195df673)"
spin-gate-v000-9k401
root@n42-poweredge-5:~# kubectl describe pod spin-gate-v000-9k401 --namespace=spinnaker
Name: spin-gate-v000-9k401
Namespace: spinnaker
Node: 172.25.30.21/172.25.30.21
Start Time: Mon, 19 Sep 2016 00:53:00 -0700
Labels: load-balancer-spin-gate=true,replication-controller=spin-gate-v000
Status: Running
IP: 172.16.33.6
Controllers: ReplicationController/spin-gate-v000
Containers:
gate:
Container ID: docker://7507c9d7c00e5834572cde2c0b0b54086288e9e30d3af161f0a1dbdf44672332
Image: quay.io/spinnaker/gate:latest
Image ID: docker://sha256:074d9616a43de8690c0a6a00345e422c903344f6876d9886f7357505082d06c7
Port: 8084/TCP
QoS Tier:
memory: BestEffort
cpu: BestEffort
State: Running
Started: Mon, 19 Sep 2016 01:14:54 -0700
Ready: False
Restart Count: 0
Readiness: http-get http://:8084/env delay=20s timeout=1s period=10s #success=1 #failure=3
Environment Variables:
Conditions:
Type Status
Ready False
Volumes:
spinnaker-config:
Type: Secret (a volume populated by a Secret)
SecretName: spinnaker-config
default-token-6irrl:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-6irrl
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
1h <invalid> 696 {kubelet 172.25.30.21} spec.containers{gate} Warning Unhealthy Readiness probe failed: Get http://172.16.33.6:8084/env: dial tcp 172.16.33.6:8084: connection refused
what's wrong here?
UPDATE1
Logs (Please check the logs here)
2016-09-20 06:49:45.062 ERROR 1 --- [ main] o.s.boot.SpringApplication : Application startup failed
org.springframework.context.ApplicationContextException: Unable to start embedded container; nested exception is org.springframework.boot.context.embedded.EmbeddedServletContainerException: Unable to start embedded Tomcat
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onRefresh(EmbeddedWebApplicationContext.java:133)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:532)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:118)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:690)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:322)
at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:134)
at org.springframework.boot.builder.SpringApplicationBuilder$run$0.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at com.netflix.spinnaker.front50.Main.main(Main.groovy:47)
Caused by: org.springframework.boot.context.embedded.EmbeddedServletContainerException: Unable to start embedded Tomcat
at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.initialize(TomcatEmbeddedServletContainer.java:99)
at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.<init>(TomcatEmbeddedServletContainer.java:76)
at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainerFactory.getTomcatEmbeddedServletContainer(TomcatEmbeddedServletContainerFactory.java:384)
at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainerFactory.getEmbeddedServletContainer(TomcatEmbeddedServletContainerFactory.java:156)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.createEmbeddedServletContainer(EmbeddedWebApplicationContext.java:159)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onRefresh(EmbeddedWebApplicationContext.java:130)
... 10 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration'........
..........
UPDATE-1(02-06-2017)
I tried above setup again in latest version of K8
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae", GitTreeState:"clean", BuildDate:"2017-05-19T18:44:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae", GitTreeState:"clean", BuildDate:"2017-05-19T18:33:17Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Still not all PODs are up
ubuntu@ip-172-31-18-78:~/spinnaker/experimental/kubernetes/simple$ kubectl get pods --namespace=spinnaker
NAME READY STATUS RESTARTS AGE
data-redis-master-v000-rzmzq 1/1 Running 0 31m
spin-clouddriver-v000-qhz97 1/1 Running 0 31m
spin-deck-v000-0sz8q 1/1 Running 0 31m
spin-echo-v000-q9xv5 1/1 Running 0 31m
spin-front50-v000-646vg 0/1 CrashLoopBackOff 10 31m
spin-gate-v000-vfvhg 0/1 Running 0 31m
spin-igor-v000-8j4r0 1/1 Running 0 31m
spin-orca-v000-ndpcx 0/1 CrashLoopBackOff 9 31m
Here is the logs links
Front50 https://pastebin.com/ge5TR4eR
Orca https://pastebin.com/wStmBtst
Gate https://pastebin.com/T8vjqL2K
Deck https://pastebin.com/kZnzN62W
Clouddriver https://pastebin.com/1pEU6V5D
Echo https://pastebin.com/cvJ4dVta
Igor https://pastebin.com/QYkHBxkr
Did I miss any configuration? I have not touched yaml config(Updated Jenkins URL,uname, passwd), that's I'm getting errors?. I'm new to Spinnaker. I had little knowledge on normal Spinnaker installation. Please guide me the installation.
Thanks
312
use halyard to install spinnaker. it is the recommended approach for deploying spinnaker in kubernetes clsuter
165
Based on these errors, it seems that the Front50 service cannot reach any backend(if there is one configured). I would not modify the spinnaker-local.yml file directly, but instead use Halyard to install Spinnaker services into Kubernetes.
I have setup Spinnaker services in Kubernetes successfully using the instructions and scripts in this repo. Just omit/skip the components that you don't need.
https://github.com/grizzthedj/kubernetes-spinnaker
33
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
