feat: swap 4949 workflow overhaul

[jekabs-karklins]

Description

This PR

• introduces new database structures to improve the workflow management system
• extends functionality by making workflows more capable
• improves UI to enable new capabilties and align with material design guidelines

Motivation and Context

The current workflow management system has limitations and inefficiencies that are addressed by the introduction of new database structures. This change is required to enhance the system's performance and user experience.

State machine

This workflow overhaul aims to align the state changes of proposal to classical state machine behaviour. The implementation of abstract state machine can be found in stateMachnine.ts. It matches the popular xState library interface just is much more simple.

There are 5 main concepts in the state machine.

• Status (State): A specific, stable stage in a proposal's lifecycle (e.g., DRAFT). The proposal stays in this state until a valid transition is triggered.
• Event: A signal or trigger that initiates a potential transition (e.g., PROPOSAL_SUBMITTED). It informs the machine that an action has occurred and a state change may be required.
• Guard: A validation or condition (e.g., isProposalInstrumentSelected) that must return true for the transition to proceed. It acts as a gatekeeper to ensure business rules are met.
• Action (Side-effect): An automated task executed during the transition itself (e.g., notifying RabbitMQ). These are the reactive behaviors that happen as a result of the state change.
• Transition: The process of moving from one status (source) to another (target). A transition is triggered by an Event, must clear any Guards, and may execute one or more Actions as it resolves.

ER diagram of the new Workflow Data Structure

Note: in order to keep the PR smaller I have not moved some tables into the DB but kept them in the code (color gray), but it makes sence to depict them here in the ER diagram.

Entity Tables:

• workflows: contains the main record for workflows containing name, description and workflow_type
• statuses: contains all statuses that the entity (proposal, exp.safety) can be in
• proposals: the propoals table
• workflow_status_connections: defines the transitions (see Transition under "State machine" section)
• worklow_status_changing_events: defines the events (see Event under "State machine" section)
• workflow_status_actions: defines the actions/side-effects (see Action under "State machine" section)
• workflow_status_changing_guards: defines the guards effects (see Guard under "State machine" section)

Changes

This PR removes a lot of unneeded code in a way simplifying the implementation

Here are the main changes to pay attention when reviewing

1. Cleanup of the namings DefaultStatus and InitialStatus.
   Initial status is the status of the entity when it enters the workflow. DRAFT for proposal and AWAITING_ESF for safetyReview.
   Defaultstatus is status which system is aware of and is not created by user with through "Create status" button. These two namings were mixed and used interchangebly, but now they are separated.

2. The table proposal_events and experiment_safety_events has been dropped and their alternative implementation has been implemented in guards which now checks if the condition is true during the transition. (see apps/backend/src/workflowEngine/guards/*.ts)

2.1 There might be an important change for STFC. Since I have removed the proposal_events table and along with it this proposalEvents.proposal_all_fap_reviews_submitted. Since this is not available any more I have changed to check if all fap reviews are submited by inspecting the fap review table columns https://github.com/UserOfficeProject/user-office-core/pull/1297/changes#diff-f3b982a00651344db8c6516a09c950e0e30df0ac84e1cd543826a0d45875588eR209

3. Status primary key now uses the semantic string statusId (formerly short_code), dropping the surrogate int. This removes number => string mapping in the code, and keeps IDs readable in logs/DB dumps and tables that reference status. This prevents mismatch bugs where int keys and short codes drifted. Migration 0205_RenameShortCodeToStatusId.sql rewires FK references across proposals/experiment_safety/workflow tables and reconstitutes dependent views.
   Operationally this makes debugging and API payloads self-describing and aligns status identity with the business identifier.

4. Status actions are now attached on transition. Before they were attached the status. This means, that if you change the status manually, the status action will not be run.

How Has This Been Tested?

Added and updated e2e tests

Fixes Jira Issue

https://jira.esss.lu.se/browse/SWAP-4949

[ACLay]

What is this PR actually trying to do? The body mentions "limitations and inefficiencies" but it seems so vague I have no idea, and I don't have access to the linked ESS jira item. If this is for the "allowing a status to be on a workflow multiple times" thing mentioned at sprint planning, I thought that was going to go to a discussion thread so we could figure out what the approach to take should be?

And from a review standpoint, I don't see how I'm practically going to be able to review 10k lines of code changes.

[jekabs-karklins]

What is this PR actually trying to do? The body mentions "limitations and inefficiencies" but it seems so vague I have no idea, and I don't have access to the linked ESS jira item. If this is for the "allowing a status to be on a workflow multiple times" thing mentioned at sprint planning, I thought that was going to go to a discussion thread so we could figure out what the approach to take should be?

And from a review standpoint, I don't see how I'm practically going to be able to review 10k lines of code changes.

Hi @ACLay ,

All valid points, and I have therefore not marked this as review:please just yet.

Side note: When started on this overhaul I did not realize that workflow touches so many things in the application, and this turned out the be the biggest PR I have made.
This PR is a bit of a special case and I was hoping we could set up the meeting as there are number of important things to talk about what this PR affects, as well get feedback from you guys on the topic.

Meanwhile I will try to improve the description of the PR to give high level overview.

Thanks for the proactive comment