EmbeddedKafka.stop() fails to shutdown Zookeeper

[thrykol]

Using "net.manub" %% "scalatest-embedded-kafka" % "1.1.1" I'm seeing an issue with stopping the embedded servers if the servers have been started individually. When I start the servers by calling start() the subsequent call to stop() works as expected. However, when starting each server individually, a call to stop() results in repeated Connection refused ZooKeeper errors .

Looking at the code, the discrepancy can be explained by the fact that start() does not add the EmbeddedZ instance to servers and is thus never stopped by a call to stop() while startZooKeeper() adds the instance to servers.

This is best demonstrated in the REPL (I canceled the session after two errors but they repeat indefinitely):

[info] Starting scala interpreter...
Welcome to Scala 2.12.6 (OpenJDK 64-Bit Server VM, Java 1.8.0_172).
Type in expressions for evaluation. Or try :help.

scala> :paste
// Entering paste mode (ctrl-D to finish)

import net.manub.embeddedkafka.EmbeddedKafka
import scala.reflect.io.Directory
import org.slf4j.LoggerFactory

val log = LoggerFactory.getLogger("REPL-logger")

log.info("EmbeddedKafka.start()")
EmbeddedKafka.start()
Thread.sleep(500)
EmbeddedKafka.stop()
Thread.sleep(500)

log.info("EmbeddedKafka.startZooKeeper()")
EmbeddedKafka.startZooKeeper(Directory.makeTemp("zookeeper-logs"))
Thread.sleep(500)

log.info("EmbeddedKafka.startKafka()")
EmbeddedKafka.startKafka(Directory.makeTemp("kafka-logs"))
Thread.sleep(500)

log.info("Stopping servers")
EmbeddedKafka.stop()
log.info("Servers stopped")

// Exiting paste mode, now interpreting.

INFO  REPL-logger - EmbeddedKafka.start()
ERROR o.a.z.s.ZooKeeperServer - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
WARN  k.s.BrokerMetadataCheckpoint - No meta.properties file under dir /tmp/kafka-logs3439956962492633719.tmp/meta.properties
WARN  k.s.BrokerMetadataCheckpoint - No meta.properties file under dir /tmp/kafka-logs3439956962492633719.tmp/meta.properties
ERROR o.a.z.s.ZooKeeperServer - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
INFO  REPL-logger - EmbeddedKafka.startZooKeeper()
ERROR o.a.z.s.ZooKeeperServer - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
INFO  REPL-logger - EmbeddedKafka.startKafka()
WARN  o.a.z.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
  at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
  at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
  at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
WARN  k.s.BrokerMetadataCheckpoint - No meta.properties file under dir /tmp/kafka-logs5375698435158631498.tmp/meta.properties
WARN  k.s.BrokerMetadataCheckpoint - No meta.properties file under dir /tmp/kafka-logs5375698435158631498.tmp/meta.properties
INFO  REPL-logger - Stopping servers
ERROR o.a.z.s.ZooKeeperServer - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
WARN  o.a.z.ClientCnxn - Session 0x164d250d4e10000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
  at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
  at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
  at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
WARN  o.a.z.ClientCnxn - Session 0x164d250d4e10000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
  at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
  at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
  at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

[thrykol]

Sorry to comment on a closed issue, but not sure how to get around it. I've just built and tested master and am still seeing the same issue. #152 didn't address the underlying issue of the process hanging when stopping ZooKeeper and Kafka after starting each service independently (startZookeeper and startKafka.

[manub]

No worries @thrykol, I'm sorry this didn't fix the problem.

Would you be able to submit a PR to fix this?

[francescopellegrini]

I think this was fixed in the new WIP repo...