Data migration from C7 to C8 RDBMS

[yanavasileva]

User Story (Required on creation)

• As a process engineer, I want to migrate the history data from C7 to C8 so that I can maintain a complete audit trail for compliance purposes.
• As a process engineer, I want to perform retrospective analysis on historical data to identify bottlenecks and improve process efficiency.
• As a process engineer, I want to access historical data after migrating from C7 to C8 to troubleshoot issues effectively and understand the context of problems.
• As an IT administrator, I want to decommission C7 and manage only the C8 system to reduce complexity and avoid exceeding Camunda 7's support period.

Functional Requirements (Required before implementation)

Data migrator project exist to server the resolve the listed user stories.
Standalone application that can be run in different modes to perform runtime, history or both migration.
User has an option to track and rerun instances that have been skip during the migration.

Technical Requirements (Required before implementation)

Follow technical proposal: https://docs.google.com/document/d/1LfVsc4NSvsRws6Dswl0jvOJB6o5O8nRZe7vUCOCo3Jg/edit?tab=t.2ii1xthvheqb#heading=h.6m6oyuyo4d2f

Limitations of Scope

Hints

Diagram converter: https://diagram-converter.consulting-sandbox.camunda.cloud/

Links

Breakdown

General

• In Data migrator, Add Camunda 8 license for the project #4982 (tackle first)
• In Data migrator, Add initial testing setup for black box testing #4983 (tackle first)
• In Data migrator, Setup CI #4984 (tackle first)
• In Data migrator, Create packaging functionality to produce distribution as a standalone application #4988
• In Data migrator, Add a mechanism to keep track of skipped entities #4985
• In Data migrator, Introduce mode to list and retry the migration of skipped entities #4986
• In Data migrator, Setup log output #4987
• In Data migrator, Extension mechanisms exist that allow users to hook into it #4989
• In Data migrator, Add release workflow #4990
• In Data migrator, Extend database coverage #4991
• In Data migrator, Documentation exist #4992

Runtime migration

• In Data migrator, Adjust C7 queries to allow migrating the runtime to C8 #4993
• In Data migrator, Make C7 and C8 queries paginated #4994
• In Data migrator (runtime), Add support for the migration of decisions and forms #4995
• In Data migrator, Add tests for all executable BPMN elements #4996
• In Data migrator, Make runtime migration idempotent #4997
• In Data migrator, Handle multiple-levels of the call hierarchy #4998
• In Data migrator, Add validation of the runtime state #4999
• In Data migrator, Make the C8 Client configurable via the application.yml #5000

History migration

• In Data migrator (history), Validate general approach for migration of entities #5001
• In Data migrator, Parametrize runtime tests #5002
• In Data migrator (history), Migrate process definition entities #5003
• In Data migrator (history), Migrate process instance entities #5004
• In Data migrator (history), Migrate flow node entities #5005
• In Data migrator (history), Migrate user task entities #5006
• In Data migrator (history), Migrate variable entities #5007
• In Data migrator (history), Migrate incident entities #5008
• In Data migrator, The migrator is fault-tolerant #5009

Nice-to-have (move to separate ticket when not tackled)

• In Data migrator, Allow to drop the MIGRATION_MAPPING table after the migration has been completed #5010
• In Data migrator, Add examples module that will demonstrate usage of the Migrator #5011
• In Data migrator (runtime), Auto-add the “migrator” listener to the model #5012
• In Data migrator (history), Improve performance with indices #5013
• In Data migrator (history), Improve flush strategy #5014
• In Data migrator, Have multiple threads to parallelize migration #5017

Dev2QA handover

• Does this ticket need a QA test and the testing goals are not clear from the description? Add a Dev2QA handover comment