[FEATURE] Allow cancelling of active long running tasks: experiment execution, judgment rating generation

[martin-gaievski]

Is your feature request related to a problem?

If user needs to interrupt active task, it's not possible in SWR, only option is either to wait task completion or restarting the cluster. Both are not great options and definitely not good user experience

What solution would you like?

Any type of control that allows me to interrupt such active task. On a backend we can interrupt all active threads and cancel all queued threads. Partial results can be deleted as part of the cancellation process as they are probably not needed in such scenario.

What alternatives have you considered?

For local cluster I can just restart it. Simple deletion of the task from UI doesn't do anything as work has been scheduled as async long running task with multiple working threads.

Do you have any additional context?

I bumped into such problem with LLM generated ratings and hybrid optimizer experiment. Example scenario:

• I started rating generation with LLM, for large dataset and number of ratings this is long running process
• after some time I started hybrid optimizer experiment that uses those LLM ratings. This is possible because document record for judgment ratings has been created right away. Note that actual rating generation is running and all ratingS are empty/0.0
• At this point I realize hybrid optimizer experiment results are not useful because of incomplete ratings, all metric will be 0.0

Problem: I cannot cancel hybrid optimizer experiment even if I know this is throw away work. I had to wait 15mins or restart cluster.

[epugh]

Makes a lot of sense. Are there any existing patterns in OpenSearch for doing this type of thing?

[martin-gaievski]

OpenSearch allows some tasks to be cancelled with usage of the API https://docs.opensearch.org/latest/api-reference/tasks/cancel-tasks/. Task must support this or being cancellable.
In case of SRW we are using custom framework for long running tasks, so same low level concepts are not automatically applicable.
I think what we probably can do is this:

• let all already started tasks (CompletableFuture instances) to finish. as I remember each task is quite small and is responsible for a single unit, like getting rating for a single document or getting search results for a single query. Interrupting running tasks can leave system in unpredictable inconsistent state, and will not benefit end-user a lot.
• all scheduled but not yet started task we can cancel
• update the status of the main task to something meaningful, like CANCELLED. I'm not in favor of using COMPLETED or ERROR, both are misleading.
  Then user most likely will delete such task from UI, this is where there is another gap (should be addressed separately), we are not deleting all the entries but only the main item. For instance, for such cancelled experiment we also need to delete variants, results and other connected records.

[epugh]

I wonder... If we had wrapped our long running tasks in the Job Scheduler framework, would that have helped? Treated everything as a job that get submitted, and then use that framework to manage the lifecycle?

I'm hoping that whatever solution we come up with in SRW, we can generalize it to other long running processes in OpenSearch so we solve it not just for this plugin!

Agreed on data integrity, I opened up #238 and have discussed this with @ajleong623 as something to tackle once we get scheduled experiments PR completed.

[ajleong623]

@epugh @martin-gaievski With the timeout in the job scheduler PR, the tasks would only be cancelled if the timeout occurs, but I believe it can be extended to an api that can cancel those long-running tasks. This can also be integrated in a potential WLM framework like the one in OpenSearch. However, with WLM, the tasks are automatically rejected if the limits are reached. I am thinking that for SRW, we can wrap the requests as a longer-running tasks that can be cancelled.