feat(23570): Add controller for workspace backup #1530
+2,285
−20
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Allda
requested review from
akurinnoy,
dkwon17,
ibuziuk and
rohanKanojia
as code owners
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: Allda The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Allda
force-pushed
the
23570
branch
2 times, most recently
from
42dd45c to
dffd7e6
Compare
Member
|
@Allda : Really appreciate you taking the time to contribute this in such a short time. 🎉 Could you please also fill out the “Is it tested? How?” section in the PR template? It’ll help reviewers and future contributors verify the change more easily. Thanks again for your effort! 🙌 |
Member
|
I tested this PR and it seems to work.
config:
workspace:
backupCronJob:
enable: true
schedule: "*/3 * * * *"
|
Allda
force-pushed
the
23570
branch
3 times, most recently
from
0bc74b1 to
8427ba5
Compare
Collaborator
Author
|
/retest |
tolusha
reviewed
Oct 29, 2025
tolusha
reviewed
Oct 29, 2025
tolusha
reviewed
Oct 29, 2025
tolusha
reviewed
Oct 29, 2025
dkwon17
reviewed
Oct 29, 2025
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1530 +/- ##
==========================================
+ Coverage 34.09% 35.30% +1.21%
==========================================
Files 160 161 +1
Lines 13348 13802 +454
==========================================
+ Hits 4551 4873 +322
- Misses 8487 8599 +112
- Partials 310 330 +20 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
tolusha
reviewed
Nov 5, 2025
tolusha
reviewed
Nov 5, 2025
tolusha
reviewed
Nov 7, 2025
Collaborator
Author
|
/retest |
A new backup controller orchestrates a backup process for workspace PVC. A new configuration option is added to DevWorkspaceOperatorConfig that enables running regular cronjob that is responsible for backup mechanism. The job executes following steps: - Find a workspaces - Finds out that workspace has been recently stopped - Detect a workspace PVC - Execute a job in the same namespace that does the backup The last step is currently not fully implemented as it requires running a buildah inside the container and it will be delivered as a separate feature. Issue: eclipse-che/che#23570 Signed-off-by: Ales Raszka <[email protected]>
A backup of workspace is done using Buildah and storing a content of the workspace PVC into a container image. The image is later stored in a registry and can be used to recover data. A prototype script was updated and stored under project-backup directory and is build alongside the controller. The backup job calls the script and execute following steps: - mount a volume with workspace data - build container image using buildah - push image to registry configured by the operator admin Signed-off-by: Ales Raszka <[email protected]>
A new sub-object was added to the operator config that reflect a current status of the backup controller and stores a last time the backup was executed. This value is used to determine whether a backup of the workspace is needed or if it already has been executed. Signed-off-by: Ales Raszka <[email protected]>
A backup job use a PVC name from a default value or from the config if user configured custom name. Signed-off-by: Ales Raszka <[email protected]>
The backup job can now push to registries which requires auth token. The token is provided as a secret in operator namespace and added to the operator config. Signed-off-by: Ales Raszka <[email protected]>
Signed-off-by: Ales Raszka <[email protected]>
A backup job now determines the name of pvc based on used storage type. It distinguish between different storage types (common and per-workspace) and mount the volume dynamically. Signed-off-by: Ales Raszka <[email protected]>
It turns out the capabilities from the prototype are not needed. Signed-off-by: Ales Raszka <[email protected]>
A new SA is created for the backup jobs to limit the permission to just what is necessary. Signed-off-by: Ales Raszka <[email protected]>
Signed-off-by: Ales Raszka <[email protected]>
- Make registry field required - Replace custom bool comparison with library - Minor tweeks Signed-off-by: Ales Raszka <[email protected]>
Use single logger across the controller and only add context if needed. Signed-off-by: Ales Raszka <[email protected]>
Collaborator
|
@Allda maybe I'm missing something but I am getting this error: This is my DWOC: Any ideas? |
Collaborator
Author
Could you please share more details about the created Job? Are there any events or logs? I tried it today with a similar config (only difference is registry location) and the push Job passed. |
Collaborator
|
Here is the job's yaml: job-devworkspace-backup-29mb2.yaml There weren't any pod logs (since the pod never started) but only a lot of the Screen.Recording.2025-11-18.at.3.59.33.PM.movOutput of |
A registry configuration is now stored under a separated nested struct. Signed-off-by: Ales Raszka <[email protected]>
A SA is created for every backup workspace to avoid ownership conflict. Signed-off-by: Ales Raszka <[email protected]>
A switching a based image to podman allowed us to run a backup job as a regular user 1000 without any privileged escalation. Signed-off-by: Ales Raszka <[email protected]>
Signed-off-by: Ales Raszka <[email protected]>
Signed-off-by: Ales Raszka <[email protected]>
|
@Allda: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Member
|
@Allda : Edit: I was able to resolve this issue by granting additional permissions to ServiceAccount , however I'm not sure whether this is required step or some issue: After updating ServiceAccount permissions, I'm able to see backup job getting executed and creating images on the configured registry: 
Question: Will the backup image will be platform dependent? I see only I'm also facing the same issue as David. I was trying to test your changes on CRC. Environment:OS: Linux CRC Version: Steps to Reproduce:
config:
workspace:
backupCronJob:
enable: true
registry:
authSecret: dockerhub-push-secret
path: docker.io/rohankanojia
schedule: '* * * * *'
# run From DWO root dir
oc create -f samples/code-latest.yaml
oc patch devworkspace code-latest \
--type=merge \
-p '{"spec": {"started": false}}'
Upon checking details I see this error: Perhaps pod creation is getting forbidden by OpenShift SecurityContextConstraints due to this: SecurityContext: &corev1.SecurityContext{
RunAsUser: ptr.To[int64](1000),
}, |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A new backup controller orchestrates a backup process for workspace PVC. A new configuration option is added to DevWorkspaceOperatorConfig that enables running regular cronjob that is responsible for backup mechanism. The job executes following steps:
The last step is currently not fully implemented as it requires running a buildah inside the container and it will be delivered as a separate feature.
Issue: eclipse-che/che#23570
What does this PR do?
What issues does this PR fix or reference?
Is it tested? How?
The feature has been tested locally and using integration tests. Following configuration should be added to the config to enable this feature:
After a config is added, stop any workspace and wait till a backup job is created.
The job creates a backup and push image to registry
PR Checklist
/test v8-devworkspace-operator-e2e, v8-che-happy-pathto trigger)v8-devworkspace-operator-e2e: DevWorkspace e2e testv8-che-happy-path: Happy path for verification integration with Che