-
Notifications
You must be signed in to change notification settings - Fork 288
DOC-6026 page about when to use RDI #2505
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 3 commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
1517b3c
DOC-6026 added RDI 'when to use' page
andy-stark-redis e4212cf
DOC-6026 improved decision tree
andy-stark-redis 1ce835d
DOC-6026 Augment's lessons learned from the tree implementation
andy-stark-redis 887a9f9
Update content/integrate/redis-data-integration/when-to-use.md
andy-stark-redis f58b9bb
DOC-6026 modify page to use embed
andy-stark-redis 1fe1fb2
Merge branch 'DOC-6026-rdi-when-to-use' of github.com:redis/docs into…
andy-stark-redis a417fe7
DOC-6026 decision tree improvements
andy-stark-redis 5cc44a7
DOC-6026 further tree improvements
andy-stark-redis e2f447c
DOC-6026 moved content to an embed
andy-stark-redis fd8681f
DOC-6026 added embed to RDI Cloud page
andy-stark-redis 4d3fb5d
Enable embed headers TOC for rc/rdi
cmilesb e91d507
DOC-6026 added tocEmbedHeaders
andy-stark-redis 701b115
DOC-6026 removed decision tree from Cloud section
andy-stark-redis File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
169 changes: 169 additions & 0 deletions
169
content/integrate/redis-data-integration/when-to-use.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,169 @@ | ||
| --- | ||
| Title: When to use RDI | ||
| alwaysopen: false | ||
| categories: | ||
| - docs | ||
| - integrate | ||
| - rs | ||
| - rdi | ||
| description: Understand when (and when not) to use RDI. | ||
| group: di | ||
| hideListLinks: false | ||
| linkTitle: When to use RDI | ||
| summary: Redis Data Integration keeps Redis in sync with the primary database in near | ||
| real time. | ||
| type: integration | ||
| weight: 5 | ||
| --- | ||
|
|
||
| RDI is designed to support apps that must use a disk based database as the system of record | ||
andy-stark-redis marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| but must also be fast and scalable. This is a common requirement for mobile and web | ||
| apps with a rapidly-growing number of users; the performance of the main database is fine at first | ||
| but it will soon struggle to handle the increasing demand without a cache. | ||
|
|
||
| ## When to use RDI | ||
|
|
||
| You should use RDI when: | ||
|
|
||
| - You must use a slow database as the system of record for the app. | ||
| - The app must always *write* its data to the slow database. | ||
| - You already intend to use Redis for the app cache. | ||
| - The data changes frequently in small increments. | ||
| - Your app can tolerate *eventual* consistency of data in the Redis cache. | ||
| - RDI throughput during | ||
| [full sync]({{< relref "/integrate/redis-data-integration/data-pipelines#pipeline-lifecycle" >}}) would not exceed 30K records per second and during | ||
| [CDC]({{< relref "/integrate/redis-data-integration/data-pipelines#pipeline-lifecycle" >}}) | ||
| would not exceed 10K records per second. | ||
|
|
||
| ## When not to use RDI | ||
|
|
||
| You should *not* use RDI when: | ||
|
|
||
| - You are migrating an existing data set into Redis only once. | ||
| - Your app needs *immediate* cache consistency rather than *eventual* consistency. | ||
| - The data is ingested from two replicas of Active-Active at the same time. | ||
| - The app must *write* data to the Redis cache, which then updates the source database. | ||
| - Your data set will only ever be small. | ||
| - Your data is updated by some batch or ETL process with long and large transactions - RDI will fail | ||
| processing these changes. | ||
|
|
||
| ## Decision tree for using RDI | ||
|
|
||
| Use the decision tree below to determine whether RDI is a good fit for your architecture: | ||
|
|
||
| ```decision-tree {id="when-to-use-rdi"} | ||
| id: when-to-use-rdi | ||
| scope: rdi | ||
| rootQuestion: systemOfRecord | ||
| questions: | ||
| systemOfRecord: | ||
| text: | | ||
| Does your app require a disk-based database as the system of record? | ||
| whyAsk: | | ||
| RDI is designed to keep Redis in sync with a primary database. If you don't need a primary database, RDI is not necessary. | ||
| answers: | ||
| yes: | ||
| value: "Yes" | ||
| nextQuestion: writeLocation | ||
| no: | ||
| value: "No" | ||
| outcome: | ||
| label: "RDI is not necessary for your use case" | ||
| id: notNecessary | ||
| sentiment: "negative" | ||
| writeLocation: | ||
| text: | | ||
| Does your app write data directly to the disk-based database? | ||
| whyAsk: | | ||
| RDI requires the primary database to be the system of record. If your app writes to Redis first, RDI won't work. | ||
| answers: | ||
| yes: | ||
| value: "Yes" | ||
| nextQuestion: consistency | ||
| no: | ||
| value: "No" | ||
| outcome: | ||
| label: "RDI won't work - your app must write to the primary database" | ||
| id: wrongWritePattern | ||
| sentiment: "negative" | ||
| consistency: | ||
| text: | | ||
| Can your app tolerate eventual consistency in the Redis cache? | ||
| whyAsk: | | ||
| RDI provides eventual consistency, not immediate consistency. If your app needs real-time cache consistency, RDI is not suitable. | ||
| answers: | ||
| yes: | ||
| value: "Yes" | ||
| nextQuestion: throughput | ||
| no: | ||
| value: "No" | ||
| outcome: | ||
| label: "RDI is not suitable - you need immediate cache consistency" | ||
| id: needsImmediate | ||
| sentiment: "negative" | ||
| throughput: | ||
| text: | | ||
| Will your throughput stay within RDI limits (≤30K records/sec during full sync, ≤10K records/sec during CDC)? | ||
| whyAsk: | | ||
| RDI has throughput limits. Exceeding these limits will cause processing failures and data loss. | ||
| answers: | ||
| yes: | ||
| value: "Yes" | ||
| nextQuestion: dataPattern | ||
| no: | ||
| value: "No" | ||
| outcome: | ||
| label: "RDI throughput limits will be exceeded" | ||
| id: exceedsLimits | ||
| sentiment: "negative" | ||
| dataPattern: | ||
| text: | | ||
| Is your data updated frequently in small increments (not by batch/ETL with large transactions)? | ||
| whyAsk: | | ||
| RDI captures changes from the database transaction log. Large batch transactions or ETL processes can cause RDI to fail. | ||
| answers: | ||
| yes: | ||
| value: "Yes" | ||
| nextQuestion: dataSize | ||
| no: | ||
| value: "No" | ||
| outcome: | ||
| label: "RDI will fail with batch/ETL processes and large transactions" | ||
| id: batchProcessing | ||
| sentiment: "negative" | ||
| dataSize: | ||
| text: | | ||
| Is your data set large enough to benefit from caching? | ||
| whyAsk: | | ||
| RDI adds operational complexity. If your data set is small, you may not need caching at all. | ||
| answers: | ||
| yes: | ||
| value: "Yes" | ||
| nextQuestion: dataSource | ||
| no: | ||
| value: "No" | ||
| outcome: | ||
| label: "RDI is not necessary - your data set is too small" | ||
| id: tooSmall | ||
| sentiment: "negative" | ||
| dataSource: | ||
| text: | | ||
| Is your data ingested from a single source (not from two Active-Active replicas simultaneously)? | ||
| whyAsk: | | ||
| RDI cannot handle data ingested from two Active-Active replicas at the same time, as this creates conflicting change events. | ||
| answers: | ||
| yes: | ||
| value: "Yes" | ||
| outcome: | ||
| label: "✅ RDI is a good fit for your use case" | ||
| id: goodFit | ||
| sentiment: "positive" | ||
| no: | ||
| value: "No" | ||
| outcome: | ||
| label: "RDI won't work with Active-Active replicas" | ||
| id: activeActive | ||
| sentiment: "negative" | ||
| ``` | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm happy to do this myself, but I'd recommend also adding this section somewhere on this page: https://redis.io/docs/latest/operate/rc/databases/rdi/ - I think it would go great before the Data Pipeline Architecture section.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@cmilesb Good idea! I don't mind doing it while I've got this PR open (it may need other changes anyway). Do you think a link to this page in the main RDI section is OK, or would you prefer to have an embed and use it in both the main section and in Cloud?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@andy-stark-redis An embed is probably the better option. Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@cmilesb The content has changed quite a bit now, but I've added all the new stuff in the Cloud page too, using an embed. Let me know what you think :-)