best-practices/graceful-shutdown

[giscus[bot]]

best-practices/graceful-shutdown

概述

https://imroc.cc/istio/best-practices/graceful-shutdown

[IceRiverDev]

大佬，想请教个问题，这里只说了针对inbound的流量，如果是outbound流量呢？我们的k8s为了满足在pod terminate阶段不会立刻中断连接，也开启了这个istio 特性。但是最近发现了一个有意思的现象：我们pod里面的entrypoint会跑三个进程，两个后台脚本进程，和一个apache进程，这两个后台脚本进程会定时去check我们的一些数据，check的过程中会跟集群外部的memcached和postgresql db进行交互，当我们手动recreate或者滚动更新这些pod的时候，一些pod会一直处于terminate的状态，直到到达最大的pod优雅关闭时间（我们设置的是10分钟），在terminate期间去查看istio sidecar的日志会发现日志里面说依然有1个active connection，但是实际上我们那个pod并没有真正的投入生产使用，因此不会存在http请求流入我们的pod，我们很费解这个活跃的连接到底是哪里来的。初步怀疑会不会是来自db或者memcached这些outbound的连接，但是我们也没办法确定。想问一下大佬会是这种原因导致的吗？另外如果我要知道istio 日志里面说的这个活跃的连接到底是哪里来的，有啥好的办法去检查吗？我最近试了好多方法都没办法确定到底是不是memcached或者db的连接。本人刚接触istio不久，对这类问题很难有好的思路去check，望大佬赐教，不胜感激。

[imroc]

也许是吧，验证方法很简单，登录到 pod 所在节点，nsenter 进入 pod 所在 netns，然后触发滚动更新，用 netstat -na|grep ESTABLISHED 看下剩余的连接对应 IP 到底是什么，反推对应的服务

[imroc]

nsenter 用法 https://imroc.cc/kubernetes/troubleshooting/skill/enter-netns-with-nsenter

[IceRiverDev]

感谢大佬的回复:), 通过以上方法和我直接在pod terminate期间进入istio sidecar容器去执行netstat，原理应该也是一样的吧? istio-proxy@tic-beta-gmp-tic-75648ff9cd-dt8wc:/$ netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 localhost:48936 localhost:15000 TIME_WAIT
tcp 0 0 localhost:37664 localhost:15000 TIME_WAIT
tcp 0 0 localhost:51470 localhost:15000 TIME_WAIT
tcp 0 0 localhost:48890 localhost:15000 TIME_WAIT
tcp 0 0 localhost:57772 localhost:15000 TIME_WAIT
tcp 0 0 localhost:57724 localhost:15000 TIME_WAIT
tcp 0 0 localhost:57752 localhost:15000 TIME_WAIT
tcp 0 0 localhost:54518 localhost:15000 TIME_WAIT
tcp 0 0 localhost:48964 localhost:15000 TIME_WAIT
tcp 0 0 localhost:33746 localhost:15020 ESTABLISHED
tcp 0 0 localhost:54506 localhost:15000 TIME_WAIT
tcp 0 0 localhost:32902 localhost:15000 TIME_WAIT
tcp 0 0 localhost:51448 localhost:15000 TIME_WAIT
tcp 0 0 localhost:32886 localhost:15000 TIME_WAIT
tcp 0 0 localhost:54484 localhost:15000 TIME_WAIT
tcp 0 0 localhost:37676 localhost:15000 TIME_WAIT
tcp 0 0 localhost:48878 localhost:15000 TIME_WAIT
tcp 0 0 localhost:40084 localhost:15000 TIME_WAIT
tcp 0 0 localhost:54544 localhost:15000 TIME_WAIT
tcp 0 0 localhost:40056 localhost:15000 TIME_WAIT
tcp 0 0 localhost:32856 localhost:15000 TIME_WAIT
tcp 0 0 localhost:57756 localhost:15000 TIME_WAIT
tcp 0 0 localhost:51440 localhost:15000 TIME_WAIT
tcp 0 0 localhost:57734 localhost:15000 TIME_WAIT
tcp 0 0 localhost:57766 localhost:15000 TIME_WAIT
tcp 0 0 localhost:40072 localhost:15000 TIME_WAIT
tcp 0 0 localhost:54482 localhost:15000 TIME_WAIT
tcp 0 0 localhost:32924 localhost:15000 TIME_WAIT
tcp 0 0 localhost:32910 localhost:15000 TIME_WAIT
tcp 0 0 localhost:48926 localhost:15000 TIME_WAIT
tcp 0 0 localhost:40054 localhost:15000 TIME_WAIT
tcp 0 0 localhost:48872 localhost:15000 TIME_WAIT
tcp 0 0 localhost:51464 localhost:15000 TIME_WAIT
tcp 0 0 localhost:40058 localhost:15000 TIME_WAIT
tcp 0 0 localhost:57732 localhost:15000 TIME_WAIT
tcp 0 0 localhost:57748 localhost:15000 TIME_WAIT
tcp 0 0 localhost:32926 localhost:15000 TIME_WAIT
tcp 0 0 localhost:32876 localhost:15000 TIME_WAIT
tcp 0 0 localhost:57730 localhost:15000 TIME_WAIT
tcp 0 0 localhost:48918 localhost:15000 TIME_WAIT
tcp 0 0 localhost:40078 localhost:15000 TIME_WAIT
tcp 0 0 localhost:32864 localhost:15000 TIME_WAIT
tcp 0 0 localhost:15001 tic-beta-gmp-tic-:42882 FIN_WAIT2
tcp 0 0 localhost:54500 localhost:15000 TIME_WAIT
tcp 0 0 localhost:40112 localhost:15000 TIME_WAIT
tcp 0 0 localhost:32940 localhost:15000 TIME_WAIT
tcp 0 0 localhost:51458 localhost:15000 TIME_WAIT
tcp 0 0 tic-beta-gmp-tic-:42896 host-10-180-180-1:11212 CLOSE_WAIT
tcp 0 0 localhost:51444 localhost:15000 TIME_WAIT
tcp 0 0 localhost:48948 localhost:15000 TIME_WAIT
tcp 0 0 localhost:51400 localhost:15000 TIME_WAIT
tcp 0 0 localhost:54556 localhost:15000 TIME_WAIT
tcp 0 0 localhost:32930 localhost:15000 TIME_WAIT
tcp 0 0 localhost:51482 localhost:15000 TIME_WAIT
tcp 0 0 localhost:40098 localhost:15000 TIME_WAIT
tcp 0 0 localhost:54560 localhost:15000 TIME_WAIT
tcp 0 0 localhost:33744 localhost:15020 ESTABLISHED
tcp 0 0 localhost:51426 localhost:15000 TIME_WAIT
tcp 0 0 localhost:45258 localhost:15090 TIME_WAIT
tcp 0 0 localhost:40096 localhost:15000 TIME_WAIT
tcp 0 0 localhost:51418 localhost:15000 TIME_WAIT
tcp 0 0 localhost:54510 localhost:15000 TIME_WAIT
tcp 0 0 localhost:40068 localhost:15000 TIME_WAIT
tcp 0 0 localhost:48902 localhost:15000 TIME_WAIT
tcp 0 0 localhost:54534 localhost:15000 TIME_WAIT
tcp 0 0 localhost:51404 localhost:15000 TIME_WAIT
tcp 0 0 localhost:48974 localhost:15000 TIME_WAIT
tcp 0 0 tic-beta-gmp-tic-:43712 istiod.istio-syst:15012 ESTABLISHED
tcp6 0 0 localhost:15020 localhost:33744 ESTABLISHED
tcp6 0 0 localhost:15020 localhost:33746 ESTABLISHED
tcp6 0 0 100.64.5.243:15020 100-64-21-218.pro:53322 ESTABLISHED
Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags Type State I-Node Path
unix 3 [ ] STREAM CONNECTED 1043173022
unix 3 [ ] STREAM CONNECTED 1043175761 ./var/run/secrets/workload-spiffe-uds/socket
unix 3 [ ] STREAM CONNECTED 1043692848 etc/istio/proxy/XDS
unix 3 [ ] STREAM CONNECTED 1043689255 这是我在pod terminate期间，在istio sidecar里面获取的netstat信息，看了下established状态的连接，我不太确定established的连接里面哪些是可疑的，大佬能帮我看一下吗？对于localhost -> localhost的连接，即使是established状态，应该也不会导致istio sidecar那边提示有活跃连接存在吧？

[imroc]

15012 端口是 xDS 连接，15020 是 metrics 采集，看起来没什么额外的连接，按理应该不会卡在 terminating

[qijianbo010]

大佬，我想请教一个问题，（在响应时，也通知客户端去关闭长连接（对于 HTTP1 响应 “Connection: close” 这个头，对于 HTTP2 响应 GOAWAY 这个帧）。）关于你说的这段有具体是怎么实现的吗？我去看了赵化冰老师的文章说这个参数只有等待30s，并没有在响应中加http header。可以帮忙详细解释一下吗？

[imroc]

Envoy 官方文档里有写“排水”的逻辑 https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/draining

By default, the HTTP connection manager filter will add “Connection: close” to HTTP1 requests, send HTTP2 GOAWAY, and terminate connections on request completion (after the delayed close period

[dong9205]

大佬，这种方式和在istio-sidecar-injector中配置preStop的命令，您认为那种方式更合适，还是说最好这二者都添加.

[ermazi]

这个参数其实没啥用啊，不如增加 terminationDrainDuration <= terminationGracePeriodSeconds 来的管用，有时候遇到长链，能卡死 :D，另外 terminationDrainDuration 默认5s 耶，即使配置了这个 EXIT_ON_ZERO_ACTIVE_CONNECTIONS，也无济于事：D

[imroc]

有 EXIT_ON_ZERO_ACTIVE_CONNECTIONS 之后，terminationDrainDuration 就没用了，两者互斥。参考 https://www.zhaohuabing.com/istio-guide/docs/best-practice/graceful-termination/

[ermazi]

原来是这样啊，my fault那如果存在长链，那就只能依赖 terminationGracePeriodSeconds 强杀了啊，不太优雅；如果是先 EXIT_ON_ZERO_ACTIVE_CONNECTIONS，再 terminationDrainDuration，再terminationGracePeriodSeconds， 是不是实际生产上更合理。