Runaway CPU consumption and queue activity

[paoloc0]

Hi. In testing, I keep running into the situation where AmazonSQSRequester creates hundreds of temporary queues, with hundreds of messages in each, and ends up in a state it seemingly can't recover from.

It's hard to figure out what's going on without getting way too well acquainted with the source code, but I've tried to produce a simple test case which can reliably reproduce the problem (or some problem anyway - hopefully not a different one):

	@Test
	public void test() throws InterruptedException {
		AmazonSQSRequesterClientBuilder
				.standard()
				.withAmazonSQS(AmazonSQSClientBuilder.defaultClient())
				.withInternalQueuePrefix("deleteme")
				.build();
		
		while (true) Thread.sleep(500);
	}

In this example, the queue deleteme does not yet exist at test start time. If I leave this test running for a while, after a few minutes, it starts spinning away, sucking significant CPU resources, and generating a huge volume of traffic to SQS (and messages sent to SQS) - here's a summarised extract of API activity over a three second period:

16:09:45.091 [SQSMessageConsumer-Thread-1] "Action=SendMessage&Version=2012
16:09:45.306 [SQSMessageConsumer-Thread-1] "Action=ListQueueTags&Version=20
16:09:45.518 [SQSMessageConsumer-Thread-1] "Action=SendMessage&Version=2012
16:09:45.558 [AmazonSQSIdleQueueDeletingClient-Thread-2] "Action=TagQueue&V
16:09:45.730 [SQSMessageConsumer-Thread-1] "Action=TagQueue&Version=2012-11
16:09:45.949 [SQSMessageConsumer-Thread-1] "Action=DeleteMessage&Version=20
16:09:46.155 [SQSMessageConsumer-Thread-1] "Action=ReceiveMessage&Version=2
16:09:46.393 [SQSMessageConsumer-Thread-1] "Action=ListQueueTags&Version=20
16:09:46.607 [SQSMessageConsumer-Thread-1] "Action=ListQueueTags&Version=20
16:09:46.818 [SQSMessageConsumer-Thread-1] "Action=SendMessage&Version=2012
16:09:47.026 [SQSMessageConsumer-Thread-1] "Action=DeleteMessage&Version=20
16:09:47.056 [SQSMessageConsumer-Thread-3] "Action=DeleteMessage&Version=20
16:09:47.234 [SQSMessageConsumer-Thread-1] "Action=ReceiveMessage&Version=2
16:09:47.263 [SQSMessageConsumer-Thread-3] "Action=ReceiveMessage&Version=2
16:09:47.679 [SQSMessageConsumer-Thread-1] "Action=ListQueueTags&Version=20
16:09:47.680 [ForkJoinPool.commonPool-worker-1] "Action=ListQueueTags&Versi
16:09:47.890 [SQSMessageConsumer-Thread-1] "Action=ListQueueTags&Version=20
16:09:47.900 [ForkJoinPool.commonPool-worker-1] "Action=ListQueueTags&Versi

The start of the runaway process seems to correspond with the first occurrence of AmazonSQSIdleQueueDeletingClient - Checking all queues begining with prefix appearing in the logs. (I guess the random initialDelay in IdleQueueSweeper causes the delay in the first occurrence of the problem.)

I wonder if #28 is not related (is not the cause)?

[paoloc0]

(Using amazon-sqs-java-temporary-queues-client v1.0.2)