[Feature Request] Mechanism to log/track long running resource intensive query impacting availability

[jainankitk]

Is your feature request related to a problem? Please describe

Today, we have Query Insights and Coordinator/Shard slow logs to track queries taking more times than the configured threshold, but both the features require query to complete before it gets tracked.

What if the query brings down the node, before it can even complete?

Describe the solution you'd like

One of the way could be to track the time for each active query, and proactively log if it has exceeded the threshold. This can potentially cause repetitive logging for the same query. But, I guess it is better to log multiple times than not log at all. Also, doing it on the coordinator slow log avoids shard level repetitive work. @sgup432 @ansjcy @dzane17 @kaushalmahi12 - What do you guys think of this approach? Open to any other suggestions.

Related component

Search:Resiliency

Describe alternatives you've considered

No response

Additional context

No response