What would you like to be added?
Add a Prometheus metric for the number of in-memory simple tokens, and log a warning as the count grows (e.g. every 10,000 tokens).
Simple tokens live in a process-local map and are scanned under lock every second. There is currently no metric or log for the size of that map, so operators cannot see unbounded growth until the member is already unavailable.
Why is this needed?
Simple tokens are not recommended for production (auth design), but they are still the default. In production we saw a cluster become unavailable when:
--auth-token-ttl was set to 315360000 (10 years), so tokens never expired.- Clients called
Authenticate on every new stream (LeaseKeepAlive / Watch) instead of reusing a token.
The in-memory map grew without bound. simpleTokenTTLKeeper.run held simpleTokensMu while iterating the whole map; Authenticate apply takes the same lock, so KV/lease traffic stalled.
Evidence:
- 60s CPU pprof: 98.96% in
auth.(*simpleTokenTTLKeeper).run (mapiternext / memhashFallback)
- Heap: ~2.5GB in
assignSimpleTokenToUser / addSimpleToken
top: etcd 101% CPU, 2.3GB RSS- Logs:
apply request took too long (3–5s vs 100ms) on authenticate, plus invalid auth token as clients retried
I am not proposing to make simple tokens production-ready. Production should still use JWT or mTLS. This is only so operators can see the map growing before the member is pinned.
What would you like to be added?
Add a Prometheus metric for the number of in-memory simple tokens, and log a warning as the count grows (e.g. every 10,000 tokens).
Simple tokens live in a process-local map and are scanned under lock every second. There is currently no metric or log for the size of that map, so operators cannot see unbounded growth until the member is already unavailable.
Why is this needed?
Simple tokens are not recommended for production (auth design), but they are still the default. In production we saw a cluster become unavailable when:
--auth-token-ttlwas set to315360000(10 years), so tokens never expired.Authenticateon every new stream (LeaseKeepAlive/ Watch) instead of reusing a token.The in-memory map grew without bound.
simpleTokenTTLKeeper.runheldsimpleTokensMuwhile iterating the whole map;Authenticateapply takes the same lock, so KV/lease traffic stalled.Evidence:
auth.(*simpleTokenTTLKeeper).run(mapiternext/memhashFallback)assignSimpleTokenToUser/addSimpleTokentop: etcd 101% CPU, 2.3GB RSSapply request took too long(3–5s vs 100ms) onauthenticate, plusinvalid auth tokenas clients retriedI am not proposing to make simple tokens production-ready. Production should still use JWT or mTLS. This is only so operators can see the map growing before the member is pinned.