feat: Orleans 3.x Migration Tooling #9351
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
benjaminpetit
approved these changes
Mar 28, 2025
ReubenBond
reviewed
Mar 28, 2025
ReubenBond
reviewed
Mar 28, 2025
src/Azure/Orleans.Persistence.AzureStorage.Migration/AzureBlobSiloBuilderExtensions.cs
Outdated
Show resolved
Hide resolved
ReubenBond
reviewed
Mar 28, 2025
| // if not - we simply fail fast with an explicit error message | ||
| _sourceStorage = (sourceStorage is IExtendedGrainStorage oldStorageEntriesController) | ||
| ? oldStorageEntriesController | ||
| : throw new ArgumentException($"Implement {nameof(IExtendedGrainStorage)} on grainStorage to support data migration.", paramName: nameof(sourceStorage)); |
There was a problem hiding this comment.
Nit, no action needed. Ideally, we should avoid local/private code symbols in messages. We don't always adhere to that, especially not in the past.
There was a problem hiding this comment.
i think IExtendedGrainStorage is not private, so it's okay to announce it in the message. But I will avoid this in future.
ReubenBond
reviewed
Mar 28, 2025
ReubenBond
reviewed
Mar 28, 2025
ReubenBond
reviewed
Mar 28, 2025
ReubenBond
reviewed
Mar 28, 2025
ReubenBond
reviewed
Mar 28, 2025
| </PropertyGroup> | ||
|
|
||
| <ItemGroup> | ||
| <PackageReference Include="Microsoft.Orleans.Serialization" Version="7.2.7" /> |
There was a problem hiding this comment.
I thought we landed on using v9.x (eg, 9.1.2)
There was a problem hiding this comment.
i used a lower version, because 9.1.2 conflicts with other packages requiring a higher version, so even more packages must be updated if 9.1.2 is picked. I decided to have less updates here due to PR size and amount of package changes even.
ReubenBond
approved these changes
Mar 28, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Migration tooling is built as an extension to Orleans3.x, and it is designed to not make a downtime of the application. It should be used temporarily to migrate data and should eventually be replaced by Orleans 7.x+ version. It has capabilities to log and save the progress of migration and control it.
It is in the nature of Orleans that part of grains is touched, used, and their state is updated during the runtime of the application. Tooling is using it by providing a new Storage implementation, which will read and write from/to 2 underlying storages (let's call them "source" and "destination"). That would allow an implicit migration of data to happen.
However, not all of the grains will be touched and updated during runtime of app, meaning there should be a way to force migration for all other grains. Therefore, there is a separate component which encapsulates this logic. It does the following:
Migration tooling is tested in a way that for some random Grain (implementation below) is written to CosmosDB in expected format, and then a sample app using latest Orleans (8 version) reads the grain from the same CosmosDB.
Details
Fundamental components implemented are
MigrationGrainStorageandDataMigrator.MigrationGrainStorageis a component to perform a runtime-migration of grains. It should be registered as a main grainStorage, which redirects the storage calls to underlying source (existing) or destination (targeting migration) storage. It has different options to control the behavior (seeGrainMigrationMode:Disabled;ReadSource_WriteBoth;ReadDestinationWithFallback_WriteBoth;ReadDestinationWithFallback_WriteDestination;ReadWriteDestination).DataMigratoris a component to perform a background or explicit migration of those grains, which are not touched during the app runtime. It will run until it ends up the migration, and it logs all the operations performed. There is backoff included and it is built to not conflicts accross the cluster.Testing
Refer to
Tester.AzureUtils.Migrationcsproj for testing of migration tooling. I was testing against my own cosmos db, and we should start the emulator for cosmos to simulate those tests.