[Jobs][Consolidation] Fix controller failed on HA mode

[cblmemo]

Fixes #7707

In managed jobs, HA recovery and update_managed_jobs_statuses should not run at the same time to avoid it being set to CONTROLLER_FAILED, while it could be just not recovered yet. We previously run the recovery script before we start the job status refresh deamon, who calls update_managed_jobs_statuses, but this function is also called in sky jobs cancel which causes the conflict. This PR fixes this issue.

Tested (run the relevant ones):

• Code formatting: install pre-commit (auto-check on commit) or bash format.sh
• Any manual or new tests for this PR (please specify below)
• All smoke tests: /smoke-test (CI) or pytest tests/test_smoke.py (local)
• Relevant individual tests: /smoke-test -k test_name (CI) or pytest tests/test_smoke.py::test_name (local)
• Backward compatibility: /quicktest-core (CI) or pytest tests/smoke_tests/test_backward_compat.py (local)

[cblmemo]

/quicktest-core
/smoke-test

[cg505]

seems like the update_managed_jobs_statuses could still begin before we get a chance to trigger ha_recovery_for_consolidation_mode, right? or do we have some protection against that?

[cblmemo]

seems like the update_managed_jobs_statuses could still begin before we get a chance to trigger ha_recovery_for_consolidation_mode, right? or do we have some protection against that?

technically yes. but very unlikely. so the update_managed_jobs_statuses is only called in 2 places:

1. the request deamon
2. cancel_jobs_by_id, which is used by sky jobs cancel

in the first case, we dont call update_managed_jobs_statuses until ha_recovery_for_consolidation_mode finish. if a user submit a sky jobs cancel so fast that it is scheduled even before the request deamon start execution, it is possible. but this is really not possible as those request deamon starts at the api server startup.