Cassandra: seed provider lists no seeds. #89

Miouge1 · 2017-08-30T14:37:12Z

Hi,

Following the instruction of the Cassandra example leads to the following error message:

The seed provider lists no seeds.
WARN  14:15:51 Seed provider couldn't lookup host cassandra-0.cassandra.default.svc.cluster.local
Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: The seed provider lists no seeds.
ERROR 14:15:51 Exception encountered during startup: The seed provider lists no seeds.

It looks like a chicken an egg situation:

Container starts
readiness probe shows cassandra-0 is not ready
Cassandra checks the seed, cassandra-0.cassandra.default.svc.cluster.local does not resolve
Container stops with fatal error

If I comment the readinessProbe then the stateful set works.

Shouldn't this use the KubernetesSeedProvider?

Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"0480917b552be33e2dba47386e51decb1a211df6", GitTreeState:"clean", BuildDate:"2017-05-10T15:48:59Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"269f928217957e7126dc87e6adfa82242bfe5b1e", GitTreeState:"clean", BuildDate:"2017-07-03T15:31:10Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

The text was updated successfully, but these errors were encountered:

cvieri · 2017-10-23T11:35:53Z

/sig storage

ekuric · 2017-11-03T18:27:26Z

I see this issue too, is there any know fix for this?

wallrj · 2017-11-16T19:55:38Z

You can set PublishNotReadyAddresses = true on the headless service.

See:

publishNotReadyAddresses, when set to true, indicates that DNS implementations must publish the notReadyAddresses of subsets for the Endpoints associated with the Service. The default value is false. The primary use case for setting this field is to use a StatefulSet's Headless Service to propagate SRV records for its Pods without respect to their readiness for purpose of peer discovery. This field will replace the service.alpha.kubernetes.io/tolerate-unready-endpoints when that annotation is deprecated and all clients have been converted to use this field.

Miouge1 · 2017-11-17T13:38:39Z

@wallrj is that also available as an annotation on the service?

wallrj · 2017-11-17T14:53:08Z

Yes. My colleague @munnerz notes that the PublishNotReadyAddresses field has only been added in Kubernetes 1.8. Use the annotation service.alpha.kubernetes.io/tolerate-unready-endpoints for backwards compatibility. See:

Vince-Cercury · 2017-11-27T02:43:25Z

I didn't face the issue but I also noticed that the KubernetesSeedProvider is not enabled in the example. Created a separate issue: #147

Eric-Fontana-Bose · 2018-01-19T15:33:15Z

This just helped me tremendously! Thanks.

pravinsinghal · 2018-03-03T17:33:08Z

I am running into the same issue

INFO [main] 2018-03-03 09:13:27,994 GossipingPropertyFileSnitch.java:64 - Loaded cassandra-topology.properties for compatibility
WARN [main] 2018-03-03 09:13:28,091 SimpleSeedProvider.java:60 - Seed provider couldn't lookup host cassandra-0.cassandra.default.svc.cluster.local
WARN [main] 2018-03-03 09:13:28,094 SimpleSeedProvider.java:60 - Seed provider couldn't lookup host cassandra-1.cassandra.default.svc.cluster.local
The seed provider lists no seeds.
Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: The seed provider lists no seeds.
ERROR [main] 2018-03-03 09:13:28,180 CassandraDaemon.java:708 - Exception encountered during startup: The seed provider lists no seeds.

Kubernetes version 1.7.9
cassandra headless service has the annotation - service.alpha.kubernetes.io/tolerate-unready-endpoints: true
tried with cassandra binary from docker hub and gcr.io/google-samples/cassandra:v13

Same issue - Seed provider could not look up cassandra---

Any suggestions how to get past this issue ?

thx.

--pravin

Vince-Cercury · 2018-03-04T21:39:33Z

@pravinsinghal

Your interpretation of the log is not correct. What is important here is

The seed provider lists no seeds.

There are different ways of specifying seeds with Cassandra. But the bottom line, seed is just a list separated by commas. So you just have to tell cassandra which ones of your nodes are seeds.
Your cassandra hasn't even tried to contact a seed node because it's saying there is none defined

By default in the example, the statefulset.yaml specifies the seed node like this:

- name: CASSANDRA_SEEDS value: "cassandra-0.cassandra.%%NAMESPACE%%.svc.cluster.local"

The next thing that happens is that the run.sh will use this value and replace it in cassandra.yaml at run time.

I'm not sure if you will have time to ssh into the cassandra node before it crashes, but you should check the value in your cassandra.yaml:

seed_provider: # Addresses of hosts that are deemed contact points. # Cassandra nodes use this list of hosts to find each other and learn # the topology of the ring. You must change this if you are running # multiple nodes! #- class_name: io.k8s.cassandra.KubernetesSeedProvider - class_name: SEED_PROVIDER parameters: # seeds is actually a comma-delimited list of addresses. # Ex: "<ip1>,<ip2>,<ip3>" - seeds: "cassandra-0.cassandra.%%NAMESPACE%%.svc.cluster.local"

(the default value is 127.0.0.1 in cassanda.yaml but run.sh will replace it.
Note that if you want to have more seed you would do:

cassandra-0.cassandra.%%NAMESPACE%%.svc.cluster.local,cassandra-1.cassandra.%%NAMESPACE%%.svc.cluster.local
It's just a comma separated list, nothing else. That's all it takes to specify a node. Cassandra knows what to do next.

pravinsinghal · 2018-03-04T22:13:56Z

I have defined CASSANDARA_SEEDS env var
this is from my yaml file:

env:
- name: CASSANDRA_SEEDS
value: cassandra-0.cassandra.default.svc.cluster.local,cassandra-1.cassandra.default.svc.cluster.local
- name: MAX_HEAP_SIZE
value: 256M
- name: HEAP_NEWSIZE
value: 100M
- name: CASSANDRA_CLUSTER_NAME
value: "Cassandra"
- name: CASSANDRA_DC
value: "DC1"
- name: CASSANDRA_RACK
value: "Rack1"
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: CASSANDRA_ENDPOINT_SNITCH
value: GossipingPropertyFileSnitch
volumeMounts:
- name: cassandra-data
mountPath: /var/lib/cassandra/data

And the log says -
WARN [main] 2018-03-03 09:13:28,091 SimpleSeedProvider.java:60 - Seed provider couldn't lookup host cassandra-0.cassandra.default.svc.cluster.local
WARN [main] 2018-03-03 09:13:28,094 SimpleSeedProvider.java:60 - Seed provider couldn't lookup host cassandra-1.cassandra.default.svc.cluster.local

So - is the lookup failing because the pod crashes before registering the endpoints ?
the yaml is a almost a copy cat from default example

Vince-Cercury · 2018-03-05T02:17:55Z

There are just warning. It would be normal that cassandra-1 is not ready.
However for cassandra-0.cassandra.default.svc.cluster.local ...

Are you sure you are using default namespace? You would need to somehow get into the pod before it crashes and see if you can ping cassandra-0.cassandra.default.svc.cluster.local .

pravinsinghal · 2018-03-05T03:25:02Z

I am using the default namespace.

kubectl get svc --namespace=default
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cassandra ClusterIP None 9042/TCP 1d

kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default cassandra-0 0/1 CrashLoopBackOff 487 1d
default cassandra-1 0/1 CrashLoopBackOff 487 1d
default cassandra-2 0/1 CrashLoopBackOff 488 1d

From kubelet logs:
Backing Off restarting container &Container{Name:cassandra,Image:cassandra,Command:[],Args:[],WorkingDir:,Ports:[{intra-node 0 7000 TCP } {tls-intra-node 0 7001 TCP } {jmx 0 7199 TCP } {cql 0 9042 TCP }],Env:[{CASSANDRA_SEEDS cassandra-0.cassandra.default.svc.cluster.local,cassandra-1.cassandra.default.svc.cluster.local nil} {MAX_HEAP_SIZE 256M nil} {HEAP_NEWSIZE 100M nil} {CASSANDRA_CLUSTER_NAME Cassandra nil} {CASSANDRA_DC DC1 nil} {CASSANDRA_RACK Rack1 nil} {POD_IP EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {CASSANDRA_ENDPOINT_SNITCH GossipingPropertyFileSnitch nil}],Resources:ResourceRequirements{Limits:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{1073741824 0} {} 1Gi BinarySI},},Requests:ResourceList{cpu: {{500 -3} {} 500m DecimalSI},memory: {{1073741824 0} {} 1Gi BinarySI},},},VolumeMounts:[{cassandra-data false /var/lib/cassandra/data } {default-token-dcbdx true /var/run/secrets/kubernetes.io/serviceaccount }],LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:&Capabilities{Add:[IPC_LOCK],Drop:[],},Privileged:nil,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[],TerminationMessagePolicy:File,} in pod cassandra-2_default(1f128c62-1ec3-11e8-a9ca-506b8d4c430a)

pravinsinghal · 2018-03-05T08:03:47Z

kube dns logs showed this

I0305 07:18:55.550159 1 dns.go:555] Could not find endpoints for service "cassandra" in namespace "default". DNS records will be created once endpoints show up.

Any idea what is happening ?

Vince-Cercury · 2018-03-05T08:06:15Z

have you create the cassandra service
kubectl get svc . What do you get

pravinsinghal · 2018-03-05T08:07:31Z

Yep the service seems to be there
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cassandra ClusterIP None 9042/TCP 47m

ekuric · 2018-03-05T08:21:33Z

@pravinsinghal
In my case I did below

kubectl create namespace cassandra
then inside this project
a) created service using .yaml as stated in this repo
b) edited cassandra-statefulset.yaml from this repo and changed to have

- name: CASSANDRA_SEEDS
            value: "cassandra-0.cassandra.cassandra.svc.cluster.local"

c) created stateful set
kubectl create -f cassandra-statefulset.yaml
Also you will need in cassandra-statefulset.yaml to adapt sections related to storageclass at end of that file.

Above worked fine in my case

]# kubectl get pods
NAME          READY     STATUS    RESTARTS   AGE
cassandra-0   1/1       Running   0          7m
cassandra-1   1/1       Running   0          6m
cassandra-2   1/1       Running   0          2m
# kubectl exec cassandra-0 -- nodetool status 
Datacenter: DC1-K8Demo
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns (effective)  Host ID                               Rack
UN  172.21.3.165  99.44 KiB  32           60.4%             a7b45c4a-9c7d-41d8-b253-6b810e5f3972  Rack1-K8Demo
UN  172.20.1.144  99.43 KiB  32           75.6%             2caf06f5-3407-4ae5-9175-ba44d79436a3  Rack1-K8Demo
UN  172.20.3.92   99.43 KiB  32           64.0%             7583ccd4-a3ff-40cb-9342-50cfbc59a614  Rack1-K8Demo

ahmetb · 2018-03-05T16:53:30Z

cc: @aledbf // this example was recently updated, is this still a problem

pravinsinghal · 2018-03-05T17:57:41Z

I am still running into this issue - even after i created the namespace.

INFO 17:53:49 Back-pressure is disabled with strategy null.
INFO 17:53:49 Unable to load cassandra-topology.properties; compatibility mode disabled
WARN 17:53:49 Seed provider couldn't lookup host cassandra-0.cassandra.cassandra.svc.cluster.local
WARN 17:53:49 Seed provider couldn't lookup host cassandra-1.cassandra.cassandra.svc.cluster.local
Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: The seed provider lists no seeds.
ERROR 17:53:49 Exception encountered during startup: The seed provider lists no seeds

this is the complete yaml

apiVersion: "apps/v1beta1"
kind: StatefulSet
metadata:
name: cassandra
namespace: cassandra
spec:
serviceName: cassandra
replicas: 3
updateStrategy:
type: RollingUpdate
selector:
matchLabels:
app: cassandra
template:
metadata:
labels:
app: cassandra
spec:
terminationGracePeriodSeconds: 1800
containers:
- name: cassandra
image: gcr.io/google-samples/cassandra:v13
imagePullPolicy: IfNotPresent
ports:
- containerPort: 7000
name: intra-node
- containerPort: 7001
name: tls-intra-node
- containerPort: 7199
name: jmx
- containerPort: 9042
name: cql
resources:
limits:
cpu: "500m"
memory: 1Gi
requests:
cpu: "500m"
memory: 1Gi
securityContext:
capabilities:
add:
- IPC_LOCK
env:
- name: CASSANDRA_SEEDS
value: "cassandra-0.cassandra.cassandra.svc.cluster.local,cassandra-1.cassandra.cassandra.svc.cluster.local"
- name: MAX_HEAP_SIZE
value: 256M
- name: HEAP_NEWSIZE
value: 100M
- name: CASSANDRA_CLUSTER_NAME
value: "Cassandra"
- name: CASSANDRA_DC
value: "DC1"
- name: CASSANDRA_RACK
value: "Rack1"
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: CASSANDRA_ENDPOINT_SNITCH
value: GossipingPropertyFileSnitch
readinessProbe:
exec:
command:
- /bin/bash
- -c
- /ready-probe.sh
initialDelaySeconds: 15
timeoutSeconds: 5
volumeMounts:
- name: cassandra-data1
mountPath: /var/lib/cassandra/data
volumeClaimTemplates:

metadata:
name: cassandra-data1
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: gold
resources:
requests:
storage: 1Gi

ekuric · 2018-03-05T18:04:22Z

@pravinsinghal

if you run cassandra pods in "cassandra" namespace, can you try what I wrote in my previous comment. From your last comment value for CASSANDRA_SEEDS is a bit strange , I guess wrong copy/paste

value: "cassandra-0.cassandra.cassandra.svc.cluster.local,cassandra-1.cassandra.cassandra.svc.cluster.local"

can you try with

            value: "cassandra-0.cassandra.cassandra.svc.cluster.local"

pravinsinghal · 2018-03-05T18:13:19Z

Yep - that was a copy/paste issue.
I changed that to single cassandra name as recommended - still the same issue

INFO 18:10:53 Initialized back-pressure with high ratio: 0.9, factor: 5, flow: FAST, window size: 2000.
INFO 18:10:53 Back-pressure is disabled with strategy null.
INFO 18:10:53 Unable to load cassandra-topology.properties; compatibility mode disabled
WARN 18:10:53 Seed provider couldn't lookup host cassandra-0.cassandra.cassandra.svc.cluster.local
The seed provider lists no seeds.
Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: The seed provider lists no seeds.
ERROR 18:10:53 Exception encountered during startup: The seed provider lists no seeds.

aledbf · 2018-03-05T19:08:10Z

@pravinsinghal please use aledbf/cassandra:v15, it's the test image for #201.
You just need to use value: "cassandra-0.cassandra.cassandra.svc.cluster.local" in the env variable

pravinsinghal · 2018-03-06T09:07:11Z

On a separate cluster created using kubeadm - I was able to successfully deploy this YAML ..

kubectl exec -ti cassandra-0 --namespace=cassandra -- nodetool status
Datacenter: DC1
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.244.1.2  89.88 KiB  32           69.5%             711c4b6d-b5e7-4184-991c-0bbdb1ef79cf  Rack1
UN  10.244.4.2  65.81 KiB  32           63.0%             e12f4027-8a86-41d5-a105-96345393d1b1  Rack1
UN  10.244.2.2  104.55 KiB  32           67.6%             8a7d9f5b-2042-4e9a-910b-f0572c7cc0fb  Rack1

On a separate cluster which is deployed by a separate mechanism (derivative of the hard way model) - this still does not work .. even with 1.9.3. Keep getting "Seed provider couldn't lookup host cassandra-0.cassandra.cassandra.svc.cluster.local".

pravinsinghal · 2018-03-07T22:17:02Z

Issue got identified.
The k8s cluster was deployed with a cluster domain - and not default settings. so the actual seed value that worked was cassandra-0.cassandra.my-namespace.svc.my-domain.

thx for all the pointers and help.

GervaisYO · 2018-03-16T23:00:02Z

Hi @pravinsinghal ,
i'm having the same problem with cassandra on a k8s Cluster with kubeadm. Can you please elabore on how you solve your problem?
It was working for me as well before. But then i change my infrastructure. I now have one master and 2 workers. The master node is part of a public subnet and of a private subnet (NAT). I startet the k8s cluster with --apiserver-advertise-address set to the private (NAT Network) ip address of the master node. The workers are part of the NAT only. Previously it was working with every nodes (master and workers) in the NAT network.

Thank you.

ahmetb · 2018-03-18T00:39:35Z

We recently updated thee cassandra example, ccing @aledbf to take a look and determine if we should close this issue.

aledbf · 2018-03-18T02:46:34Z

@GervaisYO what image are you using?

GervaisYO · 2018-03-18T10:36:06Z

I‘m using gcr.io/google-samples/cassandra:v13

fejta-bot · 2018-06-16T11:32:57Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

madireddyr · 2018-06-29T05:40:17Z

cc: @aledbf: is this image aledbf/cassandra:v15 is same as the image . cassandra:v14
could you provide docker image file for aledbf/cassandra:v15

fejta-bot · 2018-07-29T06:19:52Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

mtbbiker · 2018-08-07T11:31:38Z

Testing on PS: Kubernetes 1.10.6 in Azure AKS deployment.

Ran into the same problem with gcr.io/google-samples/cassandra:v14 also tried v13. I can get it to work with the default namespace and cassandra service name. But when I tried using a custom namespace ie cassandra the logs complain about no seeds spesified. I have also tried rebuilding the images with env settings other that the default values as spesified in the io.k8s.cassandra.KubernetesSeedProvider README.ms file in the repo path examples/cassandra/java/. I can not confirm but it seems that the KubernetesSeedProvider are unable to overwrite the default settings based on the following log

INFO  11:18:35 Unable to load cassandra-topology.properties; compatibility mode disabled
Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: io.k8s.cassandra.KubernetesSeedProvider
Fatal configuration error; unable to start server.  See log for stacktrace.
org.apache.cassandra.exceptions.ConfigurationException: io.k8s.cassandra.KubernetesSeedProvider
Fatal configuration error; unable to start server.  See log for stacktrace.
	at org.apache.cassandra.config.DatabaseDescriptor.applySeedProvider(DatabaseDescriptor.java:895)
	at org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:324)
	at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:142)
	at org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:647)
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:582)
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)

So in summary it works with the default namespace but not if you want to deploy in a custom namespace.

slavaaaaaaaaaa · 2018-08-30T18:18:33Z

Folks, I have this issue in EKS currently. I'm using qa as the namespace and cassandra-service as the service name, which should make this valid:

- name: CASSANDRA_SEEDS
  value: "cassandra-0.cassandra-service.qa.svc.cluster.local"

But at spin-up, neither cassandra-0 nor other containers in the namespace can resolve that hostname (other services resolve fine). Here's the error produced by cassandra-0:

INFO  18:52:07 Back-pressure is disabled with strategy null.
INFO  18:52:07 Unable to load cassandra-topology.properties; compatibility mode disabled
WARN  18:52:07 Seed provider couldn't lookup host cassandra-0.cassandra-service.qa.svc.cluster.local
Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: The seed provider lists no seeds.
ERROR 18:52:07 Exception encountered during startup: The seed provider lists no seeds.
The seed provider lists no seeds.

I'm using the default suggested service definition (save for publishNotReadyAddresses: true):

apiVersion: v1
kind: Service
metadata:
  name: cassandra-service
  namespace: qa
  labels:
    app: cassandra
spec:
  clusterIP: None
  ports:
    - port: 9042
  publishNotReadyAddresses: true
  selector:
    app: cassandra

Here's the service, running:

$ kubectl get svc --namespace qa
...
cassandra-service    ClusterIP    None    <none>    9042/TCP    3h
...

Here's the pod, failing:

cassandra-0                                   0/1       CrashLoopBackOff   8          19m

I've tried this with the following images, to no avail:

library/cassandra:2.1.20
gcr.io/google-samples/cassandra:v13

fejta-bot · 2018-09-29T19:26:59Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

k8s-ci-robot · 2018-09-29T19:27:06Z

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mkobit · 2018-11-21T18:35:26Z

Been running into something similar - have been trying to not make the namespace be required as part of the CASSANDRA_SEEDS but haven't gotten it to work -

        - name: CASSANDRA_SEEDS
          value: "cassandra-0.cassandra.mkobit.svc.cluster.local"
        - name: CASSANDRA_SEED_PROVIDER
          value: "io.k8s.cassandra.KubernetesSeedProvider"

fails for me with

Fatal configuration error; unable to start server.  See log for stacktrace.
org.apache.cassandra.exceptions.ConfigurationException: io.k8s.cassandra.KubernetesSeedProvider
Fatal configuration error; unable to start server.  See log for stacktrace.
        at org.apache.cassandra.config.DatabaseDescriptor.applySeedProvider(DatabaseDescriptor.java:895)
        at org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:324)
        at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:142)
        at org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:647)
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:582)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)
ERROR 18:19:26 Exception encountered during startup
org.apache.cassandra.exceptions.ConfigurationException: io.k8s.cassandra.KubernetesSeedProvider
Fatal configuration error; unable to start server.  See log for stacktrace.
        at org.apache.cassandra.config.DatabaseDescriptor.applySeedProvider(DatabaseDescriptor.java:895) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:324) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:142) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:647) [apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:582) [apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691) [apache-cassandra-3.11.2.jar:3.11.2]

This occurs when using image: gcr.io/google-samples/cassandra@sha256:7a3d20afa0a46ed073a5c587b4f37e21fa860e83c60b9c42fec1e1e739d64007. I'd really like to use the namespace without having to hardcode it into the CASSANDRA_SEEDS but not having any luck.

andrew-aernos · 2019-12-16T23:33:07Z

Fails for me when trying to create the second pod onward. Both the service and statefulsets were applied in the default namespace. I'm playing within a Ubuntu VM's single-node cluster (minikube).

minikube version

v1.6.1

kubectl version

client v1.16.3
server v1.17.0

kubectl get svc cassandra

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cassandra ClusterIP None 9042/TCP 10m

kubectl get pods

NAME READY STATUS RESTARTS AGE
cassandra-0 1/1 Running 0 7m37s
cassandra-1 0/1 CrashLoopBackOff 6 7m

kubectl logs cassandra-1

WARN  23:28:54 Seed provider couldn't lookup host cassandra-0.cassandra.default.svc.cluster.local
Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: The seed provider lists no seeds.
The seed provider lists no seeds.
ERROR 23:28:54 Exception encountered during startup: The seed provider lists no seeds.

andrew-aernos · 2019-12-18T22:43:46Z

Issue got identified.
The k8s cluster was deployed with a cluster domain - and not default settings. so the actual seed value that worked was cassandra-0.cassandra.my-namespace.svc.my-domain.

thx for all the pointers and help.

how might one check the deployed cluster domain?

imran20487 · 2020-03-11T06:14:59Z

Issue got identified.
The k8s cluster was deployed with a cluster domain - and not default settings. so the actual seed value that worked was cassandra-0.cassandra.my-namespace.svc.my-domain.

thx for all the pointers and help.

This fixed it for me. my-domain was the problem. Changed the domain and it started working.

imran20487 · 2020-03-11T06:17:09Z

Issue got identified.
The k8s cluster was deployed with a cluster domain - and not default settings. so the actual seed value that worked was cassandra-0.cassandra.my-namespace.svc.my-domain.
thx for all the pointers and help.

how might one check the deployed cluster domain?

I am working in a company which has it's down domain for kube. You may contact your local k8 administrator for the domain.

wallrj mentioned this issue Nov 16, 2017

ReadinessProbes and LivenessProbes jetstack/navigator#137

Merged

locol-media-dev mentioned this issue May 12, 2018

Cassandra Example: #235

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 16, 2018

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jul 29, 2018

k8s-ci-robot closed this as completed Sep 29, 2018

pranav-patil referenced this issue in pranav-patil/spring-kubernetes-microservices Sep 15, 2019

Updating cassandra-service configuration for kube cluster

10fb72f

hongbo-miao mentioned this issue May 16, 2022

Failed to deploy to Kubernetes cluster with hostname temporalio/helm-charts#295

Closed

Cassandra: seed provider lists no seeds. #89

Cassandra: seed provider lists no seeds. #89

Comments

Miouge1 commented Aug 30, 2017 • edited Loading

cvieri commented Oct 23, 2017

ekuric commented Nov 3, 2017

wallrj commented Nov 16, 2017

Miouge1 commented Nov 17, 2017

wallrj commented Nov 17, 2017

Vince-Cercury commented Nov 27, 2017

Eric-Fontana-Bose commented Jan 19, 2018

pravinsinghal commented Mar 3, 2018

Vince-Cercury commented Mar 4, 2018 • edited Loading

pravinsinghal commented Mar 4, 2018

Vince-Cercury commented Mar 5, 2018

pravinsinghal commented Mar 5, 2018

pravinsinghal commented Mar 5, 2018

Vince-Cercury commented Mar 5, 2018

pravinsinghal commented Mar 5, 2018

ekuric commented Mar 5, 2018 • edited Loading

ahmetb commented Mar 5, 2018

pravinsinghal commented Mar 5, 2018

ekuric commented Mar 5, 2018

pravinsinghal commented Mar 5, 2018

aledbf commented Mar 5, 2018 • edited Loading

pravinsinghal commented Mar 6, 2018 • edited by ahmetb Loading

pravinsinghal commented Mar 7, 2018 • edited Loading

GervaisYO commented Mar 16, 2018 • edited Loading

ahmetb commented Mar 18, 2018 • edited Loading

aledbf commented Mar 18, 2018

GervaisYO commented Mar 18, 2018

fejta-bot commented Jun 16, 2018

madireddyr commented Jun 29, 2018 • edited Loading

fejta-bot commented Jul 29, 2018

mtbbiker commented Aug 7, 2018 • edited Loading

slavaaaaaaaaaa commented Aug 30, 2018 • edited Loading

fejta-bot commented Sep 29, 2018

k8s-ci-robot commented Sep 29, 2018

mkobit commented Nov 21, 2018

andrew-aernos commented Dec 16, 2019 • edited Loading

andrew-aernos commented Dec 18, 2019

imran20487 commented Mar 11, 2020

imran20487 commented Mar 11, 2020

Miouge1 commented Aug 30, 2017 •

edited

Loading

Vince-Cercury commented Mar 4, 2018 •

edited

Loading

ekuric commented Mar 5, 2018 •

edited

Loading

aledbf commented Mar 5, 2018 •

edited

Loading

pravinsinghal commented Mar 6, 2018 •

edited by ahmetb

Loading

pravinsinghal commented Mar 7, 2018 •

edited

Loading

GervaisYO commented Mar 16, 2018 •

edited

Loading

ahmetb commented Mar 18, 2018 •

edited

Loading

madireddyr commented Jun 29, 2018 •

edited

Loading

mtbbiker commented Aug 7, 2018 •

edited

Loading

slavaaaaaaaaaa commented Aug 30, 2018 •

edited

Loading

andrew-aernos commented Dec 16, 2019 •

edited

Loading