Speed up FunctionAPI endpoints - specially simpler ones (NOT for release)

[JavierGOrdonnez]

NB: NOT for release

I was hoping that the "simpler" endpoints (in terms of the data they retrieve), such as retrieving just the status or just the outputs, could help the UI get some information already during the setup screen, such as suitable ranges for their parameters - and retrieving full job data takes so far prohibitively long.

To that end, I set up a test - run all three endpoints, simpler to more complex (status, outputs, everything) N times (with N = 30) and print the average time consumption. This is a simple for-loop in Python -- no delays, no frontend-backend communication, no concurrency.

Results are surprising -- the more complex systematically takes less time. And I do wonder how it is possible. In any case, would be great if all endpoints (and specially those for very specific information, such as the status and the outputs) could be much faster.

Code:

def test_job_retrieval_endpoints_speed(job_uid: str, N: int = 30):
    def _timeit(fun: Callable, N: int, *args, **kwargs):
        """Helper function to time the execution of a function N times."""
        import time
        start_time = time.time()
        for _ in range(N):
            result = fun(*args, **kwargs)
            _logger.info(f"Iteration {_+1}/{N}: {result}")   # Print the result of each iteration
        end_time = time.time()
        return (end_time - start_time) / N
    
    time_job_status = _timeit(job_api_instance.function_job_status, N, job_uid)
    time_job_outputs = _timeit(job_api_instance.function_job_outputs, N, job_uid)
    time_job_full = _timeit(job_api_instance.get_function_job, N, job_uid)
    
    _logger.debug(f"Average time to retrieve job status: {time_job_status:.4f} seconds")
    _logger.debug(f"Average time to retrieve job outputs: {time_job_outputs:.4f} seconds")
    _logger.debug(f"Average time to retrieve full job: {time_job_full:.4f} seconds")

test_job_retrieval_endpoints_speed(job_uid="aa5453be-d9e5-4e8a-a7a5-29acd113f1d2", N=30)

Results:

mmux-vite-backend-1  | 2025-06-20 15:09:50,047 - urllib3.connectionpool - DEBUG - https://api.sim4life-staging.cloud:443 "GET /v0/function_jobs/aa5453be-d9e5-4e8a-a7a5-29acd113f1d2/status HTTP/1.1" 200 20
mmux-vite-backend-1  | 2025-06-20 15:09:50,048 - flask_workflows - INFO - Iteration 1/30: status='SUCCESS'
mmux-vite-backend-1  | 2025-06-20 15:09:50,555 - urllib3.connectionpool - DEBUG - https://api.sim4life-staging.cloud:443 "GET /v0/function_jobs/aa5453be-d9e5-4e8a-a7a5-29acd113f1d2/status HTTP/1.1" 200 20
mmux-vite-backend-1  | 2025-06-20 15:09:50,555 - flask_workflows - INFO - Iteration 2/30: status='SUCCESS'
mmux-vite-backend-1  | 2025-06-20 15:09:51,038 - urllib3.connectionpool - DEBUG - https://api.sim4life-staging.cloud:443 "GET /v0/function_jobs/aa5453be-d9e5-4e8a-a7a5-29acd113f1d2/status HTTP/1.1" 200 20
mmux-vite-backend-1  | 2025-06-20 15:09:51,039 - flask_workflows - INFO - Iteration 3/30: status='SUCCESS'
...
mmux-vite-backend-1  | 2025-06-20 15:10:05,063 - flask_workflows - INFO - Iteration 30/30: status='SUCCESS'
mmux-vite-backend-1  | 2025-06-20 15:10:05,580 - urllib3.connectionpool - DEBUG - https://api.sim4life-staging.cloud:443 "GET /v0/function_jobs/aa5453be-d9e5-4e8a-a7a5-29acd113f1d2/outputs HTTP/1.1" 200 27
mmux-vite-backend-1  | 2025-06-20 15:10:05,580 - flask_workflows - INFO - Iteration 1/30: {'X+Y': 0.25262813333161427}
mmux-vite-backend-1  | 2025-06-20 15:10:06,165 - urllib3.connectionpool - DEBUG - https://api.sim4life-staging.cloud:443 "GET /v0/function_jobs/aa5453be-d9e5-4e8a-a7a5-29acd113f1d2/outputs HTTP/1.1" 200 27
mmux-vite-backend-1  | 2025-06-20 15:10:06,166 - flask_workflows - INFO - Iteration 2/30: {'X+Y': 0.25262813333161427}
...
mmux-vite-backend-1  | 2025-06-20 15:10:18,926 - flask_workflows - INFO - Iteration 30/30: {'X+Y': 0.25262813333161427}
mmux-vite-backend-1  | 2025-06-20 15:10:19,347 - urllib3.connectionpool - DEBUG - https://api.sim4life-staging.cloud:443 "GET /v0/function_jobs/aa5453be-d9e5-4e8a-a7a5-29acd113f1d2 HTTP/1.1" 200 412
mmux-vite-backend-1  | 2025-06-20 15:10:19,348 - flask_workflows - INFO - Iteration 1/30: oneof_schema_1_validator=None oneof_schema_2_validator=None oneof_schema_3_validator=None actual_instance=RegisteredProjectFunctionJob(title='Function job of function 34563613-407b-4e42-bd49-6026922b7e67', description='', function_uid='34563613-407b-4e42-bd49-6026922b7e67', inputs={'X': -1.5853108267877922, 'Y': 1.8379389601194065, 'Z': 0.35864894395403235}, outputs=None, function_class='PROJECT', uid='aa5453be-d9e5-4e8a-a7a5-29acd113f1d2', created_at=datetime.datetime(2025, 6, 12, 9, 12, 13, 776486, tzinfo=TzInfo(UTC)), project_job_id='527550a0-476d-11f0-84b9-0242ac1718f3') one_of_schemas={'RegisteredSolverFunctionJob', 'RegisteredProjectFunctionJob', 'RegisteredPythonCodeFunctionJob'} discriminator_value_class_map={}

mmux-vite-backend-1  | 2025-06-20 15:10:29,055 - flask_workflows - DEBUG - Average time to retrieve job status: 0.5197 seconds
mmux-vite-backend-1  | 2025-06-20 15:10:29,055 - flask_workflows - DEBUG - Average time to retrieve job outputs: 0.4621 seconds
mmux-vite-backend-1  | 2025-06-20 15:10:29,055 - flask_workflows - DEBUG - Average time to retrieve full job: 0.3376 seconds

[JavierGOrdonnez]

PS I tried reverting the order (first do all calls to full jobs, then to job outputs, then job status) but results are still pretty much the same:

mmux-vite-backend-1  | 2025-06-20 15:18:23,303 - flask_workflows - DEBUG - Average time to retrieve full job: 0.3335 seconds
mmux-vite-backend-1  | 2025-06-20 15:18:23,303 - flask_workflows - DEBUG - Average time to retrieve job outputs: 0.5181 seconds
mmux-vite-backend-1  | 2025-06-20 15:18:23,303 - flask_workflows - DEBUG - Average time to retrieve job status: 0.4694 seconds

[JavierGOrdonnez]

I am testing now the paginated "list_jobs_for_function_id" endpoint.
For function uid "34563613-407b-4e42-bd49-6026922b7e67", 12 jobs, it took 0.93 seconds (0.08s per items, wrt 0.33s per individual call)
For function uid "eea21c0d-6c2b-4cf4-91d1-116e6550cb22", 70 jobs, took 2.4 seconds (0.03s per item -- e.g. 10% of individual calls). At pagination limit 50 (instead of 20) took 1.53 seconds.

Therefore, pagination could be the way to go, IF it included up-to-date status & output information (which does not, at the moment). @wvangeit

[bisgaard-itis]

Thanks a lot for creating this issue @JavierGOrdonnez! This is what a Github issue should look like 😄. Very nice analysis! Let's look into, and fix this once the release is out

[wvangeit]

@JavierGOrdonnez this result is surprising, and oth not that surprising. As we have discussed before, and also something that needs to be improved after the release. When fetching a full function job, one gets the object that is stored in the db. This does not include the up-to-date status / outputs. These atm need to be retrieved explicitly with the respective endpoints. These endpoints can other sections of osparc handling osparc study jobs etc.

Obviously this needs to be improved in a sense that, if the user ask for the job info, that endpoint also should call the status/output endpoints to fetch up to date info (we can put some mechanism there to make sure it only happens if really necessary (e.g. if status is 'FINISHED' or 'FAILED' I assume it won't be necessary to update the info). Once this functionality is added, the full-job fetch will become slower than the pure status/output check.
So there will basically be two effects:

• while jobs are running, things will 'slow down' fetching that job, and also the status/output endpoints, because we now need to update the database with the up to date info
• for the finished jobs, things will get at the level of the current 'full-job' retrieval, because we only need to fetch the data from db once.

After the above is done. We can look at how to speed things up:

• use/provide pagination as much as possible
• the db access / rpc calls for these endpoints
• decrease the retrieval time of status/outputs for these jobs.

Could you however also consider making the number of calls more efficient. I mentioned it before. Atm this is still tractable, but I don't consider it scalable. Afaik atm you're pulling all the jobs, although the users might choose not to use them. I think it would be better to first show the user the job collections, let them browse the jobs (pulled by pagination based on the drop down buttons the user clicks), and 'then' pull everything the user has selected to build a SUMO from.