Patch20250220: logic optimization

[colorzzr]

Summary

This PR include several optimization:

1. change the number of concurrent job to the number of thread + 1, in order to reduce the memory usage when uploading.
2. update the logic of access_token, refresh_token and api_key logic to cache it for runtime instead of decrypt it from file everytime during commands
3. remove the outstanding token refresh thread during upload
4. optimize the logic of threading upload:
   • move finishing event chunk_upload_done into function to avoid racing condition.
   • scheduled check and timeout for finishing event chunk_upload_done.
   • so it won't run into a dead loop when resuming at 100% for chunk upload.

JIRA Issues

None

Type of Change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update
• Refactor or reformatting

Testing

Are there any new or updated tests to validate the changes?

• Yes
• No

Test Directions

(Additional instructions for how to run tests or validate functionality if not covered by unit tests)

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
__init__.py	0	0	100%
pilotcli.py	0	59	0%	5, 7–10, 12–18, 21–22, 24–25, 27–28, 30–34, 36–39, 41, 43–44, 46–47, 50, 53–57, 59, 61–64, 66–71, 74–80, 83–85
commands
__init__.py	0	0	100%
container_registry.py	0	19	59%	16, 22–24, 31–33, 41–44, 50–53, 62–65
dataset.py	0	9	89%	20, 38–39, 47, 82–83, 92, 126, 130
entry_point.py	0	11	85%	41, 47, 75, 98–100, 102–106
file.py	0	38	87%	48, 159–162, 166, 171, 174–176, 319–326, 328, 330, 344–350, 352, 381, 383, 387, 433, 436, 451–453, 459–460
folder.py	0	1	96%	23
project.py	0	2	95%	18, 61
user.py	0	9	82%	27, 44–45, 53–54, 62–63, 84–85
configs
__init__.py	0	0	100%
app_config.py	0	0	100%
config.py	0	0	100%
user_config.py	0	23	82%	35, 74, 83, 94, 127, 136–137, 140, 150, 156, 160, 164, 168, 172–173, 177, 181–182, 186, 190–191, 199, 203
utils.py	0	35	36%	13–14, 16–19, 22–23, 25, 27, 31–32, 34, 36–37, 40, 43, 45–47, 49–51, 54–57, 59, 67, 69–72, 74–75
models
__init__.py	0	0	100%
enums.py	0	0	100%
item.py	0	0	100%
service_meta_class.py	0	1	80%	9
singleton.py	0	0	100%
upload_form.py	0	4	33%	28, 40–42
resources
custom_error.py	0	0	100%
custom_help.py	0	0	100%
services
__init__.py	0	0	100%
services/clients
__init__.py	0	0	100%
base_auth_client.py	0	1	96%	54
base_client.py	0	2	95%	53, 104
services/container_registry_manager
container_registry_manager.py	0	108	17%	19–20, 23–26, 29–33, 36–44, 48–60, 62–64, 68–82, 84–86, 90–99, 101–103, 107–110, 125–136, 140–143, 147–162, 164, 166–169
services/crypto
__init__.py	0	0	100%
crypto.py	0	19	40%	35, 43–47, 49, 59–61, 69–75, 77, 79
services/dataset_manager
dataset_detail.py	0	22	71%	38, 59, 66–71, 73–75, 77–86, 88
dataset_download.py	0	22	82%	53, 63–65, 74–76, 92, 94–95, 97–98, 100, 102–104, 112–115, 149, 164
dataset_list.py	0	8	81%	36–41, 43, 52
model.py	0	0	100%
services/file_manager
__init__.py	0	0	100%
file_list.py	0	12	85%	60–66, 86–88, 100, 102
file_manifests.py	0	80	31%	18–23, 43–47, 51–57, 59, 63–68, 70, 74–75, 79–83, 85, 88, 90–91, 93–94, 96–99, 104–110, 115–120, 123–129, 131–133, 136–142, 144–146, 148–151
file_tag.py	0	40	31%	23–24, 28–32, 34, 37–49, 51–53, 55–56, 60–62, 65–69, 71–74, 76–77
services/file_manager/file_download
__init__.py	0	0	100%
download_client.py	0	105	55%	52–61, 79–80, 122–126, 128–130, 142–143, 148–149, 153–162, 165–170, 180–181, 212, 217, 223–233, 235–236, 239–240, 245–246, 250–255, 259–260, 263–265, 267–278, 285, 290, 305, 309–311, 313–326, 328
model.py	0	0	100%
services/file_manager/file_metadata
__init__.py	0	0	100%
file_metadata_client.py	0	3	95%	98–100
folder_client.py	0	4	91%	50, 79–81
services/file_manager/file_move
__init__.py	0	0	100%
file_move_client.py	0	37	59%	60–64, 66, 74, 79–83, 86–94, 96, 98–103, 114, 116–120, 122, 172–173
services/file_manager/file_trash
__init__.py	0	0	100%
file_trash_client.py	0	2	96%	69, 90
utils.py	0	3	92%	39, 63, 70
services/file_manager/file_upload
__init__.py	0	0	100%
exception.py	0	0	100%
file_upload.py	0	28	85%	38–45, 103–105, 117, 146–147, 156, 161, 197, 220–222, 258–259, 261–263, 339, 342, 346
models.py	0	2	96%	24, 44
upload_client.py	0	48	75%	103–106, 146, 218, 230–231, 233–247, 249, 251–255, 257–261, 263–264, 358–360, 430, 436–437, 442–445, 447–448
upload_validator.py	0	11	76%	46–52, 55–56, 61, 63
services/logger_services
__init__.py	0	0	100%
debugging_log.py	0	0	100%
log_functions.py	0	0	100%
services/output_manager
__init__.py	0	0	100%
error_handler.py	0	0	100%
help_page.py	0	7	91%	13–17, 19, 21
message_handler.py	0	56	73%	24, 46, 51, 66–68, 73, 83, 88, 101, 106, 113, 118, 123, 127, 143, 178, 188, 197, 215, 237, 281–292, 303–308, 310–316, 327, 331–332, 336–337, 339–340, 344, 348, 352
services/project_manager
__init__.py	0	0	100%
project.py	0	6	84%	40, 46–47, 49–51
services/user_authentication
__init__.py	0	0	100%
decorator.py	0	2	93%	27, 30
token_manager.py	0	8	88%	42–46, 73–74, 91
user_login_logout.py	0	16	84%	29–30, 41, 126, 130–136, 138–140, 144–145
utils
__init__.py	0	0	100%
aggregated.py	0	27	85%	57, 67–69, 83–85, 117, 131–132, 145, 169–176, 178–179, 207, 215, 232–233, 248, 276
TOTAL	3665	890	75%