What is the bug?

Local indices are retained longer than expected. For example, we observed a domain containing local indices for the past 3 days when delete_after_days=1.

This is caused by 2 factors:

1. Periodic job to delete indices is scheduled at a 24 hours interval starting from when the domain/node is created. If a node boots up at 14:30, the job will run at 14:30 every day in the future. We have no control over when the OpenSearch process starts, so this is non deterministic.
   

   query-insights/src/main/java/org/opensearch/plugin/insights/core/service/QueryInsightsService.java

   Lines 581 to 587 in a876495

   	scheduledFutures.add(
   	threadPool.scheduleWithFixedDelay(
   	this::deleteExpiredTopNIndices,
   	new TimeValue(1, TimeUnit.DAYS), // Check for deletable indices once per day
   	QueryInsightsSettings.QUERY_INSIGHTS_EXECUTOR
   	)
   	);

2. Expiration is calculated with current time. That means the local index creation time in relation to the daily job start time can add an extra day of retention.
   

   query-insights/src/main/java/org/opensearch/plugin/insights/core/service/QueryInsightsService.java

   Lines 649 to 651 in a876495

   	final long expirationMillisLong = System.currentTimeMillis() - TimeUnit.DAYS.toMillis(
   	((LocalIndexExporter) topQueriesExporter).getDeleteAfter()
   	);

How can one reproduce the bug?

Steps to reproduce the behavior.

What is the expected behavior?

Do not allow local indices to overstay the delete_after_days setting value

What is your host/environment?

Operating system, version.

Do you have any screenshots?

If applicable, add screenshots to help explain your problem.

Do you have any additional context?

Add any other context about the problem.