Modify the mechanism to pause indexing

[ankikuma]

This PR changes the mechanism to pause indexing which was introduced in #127173.

The original PR caused IndexStatsIT#testThrottleStats to fail. See #126359.

[ankikuma]

The problem with the original implementation of the pause lock mechanism was as follows:

1. First note that the EngineConcurrentMergeScheduler#beforeMerge and EngineConcurrentMergeScheduler#afterMerge can be called concurrently by 2 different merge threads.

2. Also note that ThreadPoolMergeScheduler#checkMergeTaskThrottling can be called concurrently from 2 different threads via ThreadPoolMergeScheduler#mergeTaskDone and ThreadPoolMergeScheduler#submitNewMergeTask

3. Next note that the above concurrency can cause Engine#IndexThrottle#activate and Engine#IndexThrottle#deactivate to be called concurrently. The IndexStatsIT#testThrottleStats test exposed that this concurrency was not handled correctly by the pause mechanism in place.

This is because the semaphore based approach relies on acquiring and releasing precise number of permits during activate and deactivate, which is not possible to synchronize. Let me try to explain with the the following scenario:
a. Thread 1 : activates throttling on shard --> acquires all but 1 permits
b. Multiple indexing jobs arrive, all waiting to acquire the single available permit --> only one goes through at a time (all is good so far)
c. Thread 2 : deactivate throttling on shard --> releases all but 1 permits --> all is good but note that the jobs in step (b) will still need to acquire and release these permits.
d. Thread 3: activates throttling immediately after the deactivate in step (c) but it is not able to acquire the requested number of permits immediately because the indexing threads in step (b) have acquired permits in the meantime to work on indexing. So the activate is waiting for permits before it can switch the lock to pauseLock.
e. Thread 4: deactivates throttling immediately following step (d) and finds that we still have a NOOP lock and asserts. In any case, it cannot release the precise number of permits deactivate would like to release, because activate hasn't even acquired them yet.

[elasticsearchmachine]

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

[elasticsearchmachine]

Hi @ankikuma, I've created a changelog YAML for you.