Cassandra Example:

[locol-media-dev]

I am encountering this problem when loading v14 of the example. I don' know if this relates to #89

Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: io.k8s.cassandra.KubernetesSeedProvider
Fatal configuration error; unable to start server. See log for stacktrace.
org.apache.cassandra.exceptions.ConfigurationException: io.k8s.cassandra.KubernetesSeedProvider

Attached is the startup log.

logs-from-cassandra-in-cassandra-0.txt

[migscabral]

Getting the same error. Just followed the steps in https://kubernetes.io/docs/tutorials/stateful-application/cassandra/

Here's the startup log:

Starting Cassandra on 10.52.22.5
CASSANDRA_CONF_DIR /etc/cassandra
CASSANDRA_CFG /etc/cassandra/cassandra.yaml
CASSANDRA_AUTO_BOOTSTRAP true
CASSANDRA_BROADCAST_ADDRESS 10.52.22.5
CASSANDRA_BROADCAST_RPC_ADDRESS 10.52.22.5
CASSANDRA_CLUSTER_NAME K8Demo
CASSANDRA_COMPACTION_THROUGHPUT_MB_PER_SEC
CASSANDRA_CONCURRENT_COMPACTORS
CASSANDRA_CONCURRENT_READS
CASSANDRA_CONCURRENT_WRITES
CASSANDRA_COUNTER_CACHE_SIZE_IN_MB
CASSANDRA_DC DC1-K8Demo
CASSANDRA_DISK_OPTIMIZATION_STRATEGY ssd
CASSANDRA_ENDPOINT_SNITCH SimpleSnitch
CASSANDRA_GC_WARN_THRESHOLD_IN_MS
CASSANDRA_INTERNODE_COMPRESSION
CASSANDRA_KEY_CACHE_SIZE_IN_MB
CASSANDRA_LISTEN_ADDRESS 10.52.22.5
CASSANDRA_LISTEN_INTERFACE
CASSANDRA_MEMTABLE_ALLOCATION_TYPE
CASSANDRA_MEMTABLE_CLEANUP_THRESHOLD
CASSANDRA_MEMTABLE_FLUSH_WRITERS
CASSANDRA_MIGRATION_WAIT 1
CASSANDRA_NUM_TOKENS 32
CASSANDRA_RACK Rack1-K8Demo
CASSANDRA_RING_DELAY 30000
CASSANDRA_RPC_ADDRESS 0.0.0.0
CASSANDRA_RPC_INTERFACE
CASSANDRA_SEEDS cassandra-0.cassandra.default.svc.cluster.local
CASSANDRA_SEED_PROVIDER io.k8s.cassandra.KubernetesSeedProvider
changed ownership of '/etc/cassandra/jvm.options' from root to cassandra
changed ownership of '/etc/cassandra/cassandra-env.sh' from root to cassandra
changed ownership of '/etc/cassandra/logback.xml' from root to cassandra
changed ownership of '/etc/cassandra/cassandra.yaml' from root to cassandra
changed ownership of '/etc/cassandra/cassandra-rackdc.properties' from root to cassandra
changed ownership of '/etc/cassandra' from root to cassandra
OpenJDK 64-Bit Server VM warning: Cannot open file /usr/local/apache-cassandra-3.11.2/logs/gc.log due to No such file or directory

INFO  08:45:20 Configuration location: file:/etc/cassandra/cassandra.yaml
INFO  08:45:21 Node configuration:[allocate_tokens_for_keyspace=null; authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_bootstrap=true; auto_snapshot=true; back_pressure_enabled=false; back_pressure_strategy=null; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=10.52.22.5; broadcast_rpc_address=10.52.22.5; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=0; client_encryption_options=<REDACTED>; cluster_name=K8Demo; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=/cassandra_data/commitlog; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories=[Ljava.lang.String;@275710fc; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_materialized_views=true; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=null; endpoint_snitch=GossipingPropertyFileSnitch; file_cache_round_up=null; file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=/cassandra_data/hints; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=all; internode_recv_buff_size_in_bytes=0; internode_send_buff_size_in_bytes=0; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=10.52.22.5; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=0; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=32; otc_backlog_expiration_interval_ms=200; otc_coalescing_enough_coalesced_messages=8; otc_coalescing_strategy=DISABLED; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=CassandraRoleManager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.OHCProvider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=/cassandra_data/saved_caches; seed_provider=io.k8s.cassandra.KubernetesSeedProvider{seeds=cassandra-0.cassandra.default.svc.cluster.local}; server_encryption_options=<REDACTED>; slow_query_log_timeout_in_ms=500; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_keep_alive_period_in_secs=300; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.TransparentDataEncryptionOptions@525f1e4e; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000]
INFO  08:45:21 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO  08:45:21 Global memtable on-heap threshold is enabled at 128MB
INFO  08:45:21 Global memtable off-heap threshold is enabled at 128MB
WARN  08:45:21 Small commitlog volume detected at /cassandra_data/commitlog; setting commitlog_total_space_in_mb to 487.  You can override this in cassandra.yaml
WARN  08:45:22 Only 1.900GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
INFO  08:45:22 Initialized back-pressure with high ratio: 0.9, factor: 5, flow: FAST, window size: 2000.
INFO  08:45:22 Back-pressure is disabled with strategy null.
INFO  08:45:22 Unable to load cassandra-topology.properties; compatibility mode disabled
Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: io.k8s.cassandra.KubernetesSeedProvider
Fatal configuration error; unable to start server.  See log for stacktrace.
org.apache.cassandra.exceptions.ConfigurationException: io.k8s.cassandra.KubernetesSeedProvider
Fatal configuration error; unable to start server.  See log for stacktrace.
	at org.apache.cassandra.config.DatabaseDescriptor.applySeedProvider(DatabaseDescriptor.java:895)
	at org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:324)
	at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:142)
	at org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:647)
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:582)
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)
ERROR 08:45:22 Exception encountered during startup
org.apache.cassandra.exceptions.ConfigurationException: io.k8s.cassandra.KubernetesSeedProvider
Fatal configuration error; unable to start server.  See log for stacktrace.
	at org.apache.cassandra.config.DatabaseDescriptor.applySeedProvider(DatabaseDescriptor.java:895) ~[apache-cassandra-3.11.2.jar:3.11.2]
	at org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:324) ~[apache-cassandra-3.11.2.jar:3.11.2]
	at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:142) ~[apache-cassandra-3.11.2.jar:3.11.2]
	at org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:647) [apache-cassandra-3.11.2.jar:3.11.2]
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:582) [apache-cassandra-3.11.2.jar:3.11.2]
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691) [apache-cassandra-3.11.2.jar:3.11.2]

[locol-media-dev]

I just pointed this yaml to the official Cassandra image and it came up fine.

[madireddyr]

@mikep-locol-media : Thank you for the info, we are using official Cassandra image but we got below error
WARN [main] 2018-06-13 18:52:18,929 SimpleSeedProvider.java:60 - Seed provider couldn't lookup host cassandra-0.cassandra.default.svc.cluster.local
Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: The seed provider lists no seeds.
The seed provider lists no seeds.
ERROR [main] 2018-06-13 18:52:19,011 CassandraDaemon.java:708 - Exception encountered during startup: The seed provider lists no seeds.

could you please let us know what SEED Configuration you have used

[locol-media-dev]

Looks like it could not find cassandra-0.cassandra.default.svc.cluster.local. Is this right? Are you running under the default namespace? Unfortunately I am in the hospital today otherwise I can put up my yaml files.

[madireddyr]

yes , i am using default namespace

ubuntu@xyz:~/cluster-deployment/kubernetes$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default cassandra-1-jd6kl 0/1 CrashLoopBackOff 7 18m

tomorrow is fine , get well

[locol-media-dev]

Ah. Is your pod cassandra-1? The url is looking for cassandra-0

[locol-media-dev]

Try changing the seed url to -1

[madireddyr]

i changed pod to Cassandra-0, still no luck . i have used exact same yaml's and both v14 & apache cass 3.11 no luck

when i try to bring up it gets stuck at pending state

ubuntu@ip-xyz:~/cluster-deployment/kubernetes$ kubectl get pods
NAME READY STATUS RESTARTS AGE
cassandra-0 0/1 Pending 0 2m

I0613 21:20:13.973078 11972 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, /" -H "User-Agent: kubectl/v1.10.3 (linux/amd64) kubernetes/2bba012" https://A89CFD1F959FBD470AF412E2D2563BCE.yl4.us-west-2.eks.amazonaws.com/api/v1/namespaces/default/pods/cassandra-0
I0613 21:20:14.351085 11972 round_trippers.go:405] GET https://A89CFD1F959FBD470AF412E2D2563BCE.yl4.us-west-2.eks.amazonaws.com/api/v1/namespaces/default/pods/cassandra-0 200 OK in 377 milliseconds
I0613 21:20:14.351222 11972 round_trippers.go:411] Response Headers:
I0613 21:20:14.351310 11972 round_trippers.go:414] Content-Type: application/json
I0613 21:20:14.351381 11972 round_trippers.go:414] Content-Length: 3049
I0613 21:20:14.351465 11972 round_trippers.go:414] Date: Wed, 13 Jun 2018 21:20:14 GMT
I0613 21:20:14.351556 11972 request.go:874] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"cassandra-0","generateName":"cassandra-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/cassandra-0","uid":"25aed3b1-6f4f-11e8-b611-0aeb33f03c92","resourceVersion":"921868","creationTimestamp":"2018-06-13T21:17:16Z","labels":{"app":"cassandra","controller-revision-hash":"cassandra-666c99545d","statefulset.kubernetes.io/pod-name":"cassandra-0"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"StatefulSet","name":"cassandra","uid":"25ab43c9-6f4f-11e8-b611-0aeb33f03c92","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"cassandra-data","persistentVolumeClaim":{"claimName":"cassandra-data-cassandra-0"}},{"name":"default-token-dwfnf","secret":{"secretName":"default-token-dwfnf","defaultMode":420}}],"containers":[{"name":"cassandra","image":"library/cassandra:3.11","ports":[{"name":"intra-node","containerPort":7000,"protocol":"TCP"},{"name":"tls-intra-node","containerPort":7001,"protocol":"TCP"},{"name":"jmx","containerPort":7199,"protocol":"TCP"},{"name":"cql","containerPort":9042,"protocol":"TCP"}],"env":[{"name":"MAX_HEAP_SIZE","value":"512M"},{"name":"HEAP_NEWSIZE","value":"100M"},{"name":"CASSANDRA_SEEDS","value":"cassandra-0.cassandra.default.svc.cluster.local"},{"name":"CASSANDRA_CLUSTER_NAME","value":"K8Demo"},{"name":"CASSANDRA_DC","value":"DC1-K8Demo"},{"name":"CASSANDRA_RACK","value":"Rack1-K8Demo"},{"name":"CASSANDRA_SEED_PROVIDER","value":"io.k8s.cassandra.KubernetesSeedProvider"},{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}}],"resources":{"limits":{"cpu":"500m","memory":"1Gi"},"requests":{"cpu":"500m","memory":"1Gi"}},"volumeMounts":[{"name":"cassandra-data","mountPath":"/cassandra_data"},{"name":"default-token-dwfnf","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"readinessProbe":{"exec":{"command":["/bin/bash","-c","/ready-probe.sh"]},"initialDelaySeconds":15,"timeoutSeconds":5,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"lifecycle":{"preStop":{"exec":{"command":["/bin/sh","-c","nodetool drain"]}}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always","securityContext":{"capabilities":{"add":["IPC_LOCK"]}}}],"restartPolicy":"Always","terminationGracePeriodSeconds":1800,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","securityContext":{},"hostname":"cassandra-0","subdomain":"cassandra","schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}]},"status":{"phase":"Pending","conditions":[{"type":"PodScheduled","status":"False","lastProbeTime":null,"lastTransitionTime":"2018-06-13T21:17:16Z","reason":"Unschedulable","message":"pod has unbound PersistentVolumeClaims (repeated 4 times)"}],"qosClass":"Guaranteed"}}
I0613 21:20:14.418950 11972 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, /" -H "User-Agent: kubectl/v1.10.3 (linux/amd64) kubernetes/2bba012" https://A89CFD1F959FBD470AF412E2D2563BCE.yl4.us-west-2.eks.amazonaws.com/api/v1/namespaces/default/pods/cassandra-0/log
I0613 21:20:14.423617 11972 round_trippers.go:405] GET https://A89CFD1F959FBD470AF412E2D2563BCE.yl4.us-west-2.eks.amazonaws.com/api/v1/namespaces/default/pods/cassandra-0/log 204 No Content in 4 milliseconds
I0613 21:20:14.423674 11972 round_trippers.go:411] Response Headers:
I0613 21:20:14.423714 11972 round_trippers.go:414] Date: Wed, 13 Jun 2018 21:20:14 GMT

[madireddyr]

if possible could you provide aws storage class example

[madireddyr]

@mikep-locol-media : we were able to resolve the ebs volume issue but we still facing same error as above ("seed provider error ") : could you send your yaml you have used for Apache Cassandra Image .

INFO 19:45:59 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO 19:45:59 Global memtable on-heap threshold is enabled at 128MB
INFO 19:45:59 Global memtable off-heap threshold is enabled at 128MB
WARN 19:45:59 Only 39.201GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
INFO 19:45:59 Initialized back-pressure with high ratio: 0.9, factor: 5, flow: FAST, window size: 2000.
INFO 19:45:59 Back-pressure is disabled with strategy null.
INFO 19:45:59 Unable to load cassandra-topology.properties; compatibility mode disabled
Exception (org.apache.cassandra.exceptions.ConfigurationException) encountered during startup: io.k8s.cassandra.KubernetesSeedProvider
Fatal configuration error; unable to start server. See log for stacktrace.
org.apache.cassandra.exceptions.ConfigurationException: io.k8s.cassandra.KubernetesSeedProvider
Fatal configuration error; unable to start server. See log for stacktrace.
at org.apache.cassandra.config.DatabaseDescriptor.applySeedProvider(DatabaseDescriptor.java:895)
at org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:324)
at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:142)
at org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:647)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:582)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691)
ERROR 19:45:59 Exception encountered during startup
org.apache.cassandra.exceptions.ConfigurationException: io.k8s.cassandra.KubernetesSeedProvider
Fatal configuration error; unable to start server. See log for stacktrace.
at org.apache.cassandra.config.DatabaseDescriptor.applySeedProvider(DatabaseDescriptor.java:895) ~[apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:324) ~[apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:142) ~[apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:647) [apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:582) [apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691) [apache-cassandra-3.11.2.jar:3.11.2]

[locol-media-dev]

cassandra.zip

Hi.. these are my yaml files. Notes:

1. I have installed on gke and on dockers for mac with K8s, include are the storage yamls for each of those. Have not tried on AWS
2. In my environment, I created a "cassandra" namespace. the service and statefulset yamls reflect that
3. Can't remember why I changed the apiVersion. Vaguely think it's because of the version of k8s on my gke cluster.
4. I run 3 replicates on Docker but for now only one replicate on gke. Just change that to whatever you want.

[madireddyr]

@mikep-locol-media : thank you very much i will give a try .
based on #89
#201
i tried above example with aledbf/cassandra:v15 and it worked . not sure whats the difference between
gcr.io/google-samples/cassandra:v14 & aledbf/cassandra:v15

will check cassandra image as well

[locol-media-dev]

Yes, I thought of trying aledbf/cassandra:v15. Just thought that switching over to the official cassandra was a better long term switch.

[madireddyr]

@mikep-locol-media : i was able to bring official cassandra , but during bootsrap few nodes restarted, but they joined the ring eventually
have you noticed this kind of behavior ,
since then they are running properly
here is the log for restarted nodes --any reason for these restarts

INFO [main] 2018-06-14 23:41:06,177 StorageService.java:1449 - JOINING: waiting for schema information to complete
ERROR [main] 2018-06-14 23:41:07,177 MigrationManager.java:193 - Migration task failed to complete
ERROR [main] 2018-06-14 23:41:08,178 MigrationManager.java:193 - Migration task failed to complete
ERROR [main] 2018-06-14 23:41:09,178 MigrationManager.java:193 - Migration task failed to complete
INFO [main] 2018-06-14 23:41:09,973 StorageService.java:1449 - JOINING: schema complete, ready to bootstrap
INFO [main] 2018-06-14 23:41:09,973 StorageService.java:1449 - JOINING: waiting for pending range calculation
INFO [main] 2018-06-14 23:41:09,973 StorageService.java:1449 - JOINING: calculation complete, ready to bootstrap
INFO [main] 2018-06-14 23:41:09,973 StorageService.java:1449 - JOINING: getting bootstrap token

[locol-media-dev]

I find a lot of k8s stuff takes a bit of time to get themselves sorted out starting. Have not dug into it.

[madireddyr]

@mikep-locol-media : could you please share github project link for your controller file
for open source apache kuberenets kops cluster .

[locol-media-dev]

Sorry, which controller file? I uploaded my cassandra yaml files in my previous post. I do not have it in a github repo. I have k8s clusters on GKE and also on Docker for Mac, if that is relevent.

[madireddyr]

@mikep-locol-media : one last question any idea how below seed provider class is being referenced in your controller cassandra-statefulset.yaml .is this something built into .
Open source Apache docker image or its a feature with KOPS cluster
as i am not able to figure out exactly how this seed provider is working with Apache Cassandra /KOPS

image: cassandra

- name: CASSANDRA_SEED_PROVIDER
value: io.k8s.cassandra.KubernetesSeedProvider

https://hub.docker.com/_/cassandra/

https://github.com/docker-library/cassandra/blob/01786683a1b060f813bce5539b1743695f9cc043/3.11/Dockerfile

appreciate your help

[madireddyr]

@mikep-locol-media : just to add above i would like to know if below particular jar file is embedded in Cassandra Image
Open source Apache Cassandra Image with KOPS is working for us but wanted to know how
its working .
https://github.com/kubernetes/examples/blob/master/cassandra/java/src/main/java/io/k8s/cassandra/KubernetesSeedProvider.java

[Bfoster-melrok]

I ran into this same issue.
Having CASSANDRA_SEED_PROVIDER set to io.k8s.cassandra.KubernetesSeedProvider threw an error on startup.

I solved this by creating my own image from gcr.io/google-samples/cassandra:v14

with 1 change:
mv /kubernetes-cassandra.jar /usr/local/apache-cassandra-3.11.2/lib/

I also had to give the service account the pod was using read access to endpoints in the namespace.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mkobit]

@Bfoster-melrok it seems like you shouldn't have to do what you suggested in #235 (comment) but that seems like an ok workaround. When I was trying to investigate what was going on yesterday, it doesnt seem like the exported CLASSPATH=/kubernetes-cassandra.jar is used anywhere.