MOLT User Scripts #22088
MOLT User Scripts #22088
Conversation
Files changed:
|
✅ Deploy Preview for cockroachdb-api-docs canceled.
|
✅ Deploy Preview for cockroachdb-interactivetutorials-docs canceled.
|
✅ Netlify Preview
To edit notification comments on pull requests, go to your Netlify project configuration. |
taroface
force-pushed
the
molt-user-scripts
branch
from
bb46e23 to
806b0c0
Compare
taroface
force-pushed
the
molt-user-scripts
branch
from
806b0c0 to
d5a73b7
Compare
…molt-user-scripts
…molt-user-scripts
…molt-user-scripts
bsanchez-the-roach
requested review from
noelcrl and
taroface
and removed request for
taroface
tuansydau
left a comment
tuansydau
left a comment
There was a problem hiding this comment.
Really good stuff - this was a beefy one :S
For how complicated and vague the Userscript API is, I think this PR does a really good job breaking down the API with good brevity without sacrificing too much detail. I've left some comments, most are nits or addressing things I should have made clearer in the drafting phase (sorry about that).
Thanks to everyone involved, insane effort on this one - I can formally approve after all the comments are addressed
noelcrl
left a comment
noelcrl
left a comment
There was a problem hiding this comment.
Thank you so much for the detailed work here! Really appreciate the care that went into this along with the insightful clarification questions along the way.
I think this PR is looking good overall, just some remaining questions/comments but nothing too against the overall main structure. Let me know if you need more info.
Co-authored-by: Noel Riopel <noel.riopel@cockroachlabs.com>
…molt-user-scripts
rmloveland
left a comment
rmloveland
left a comment
There was a problem hiding this comment.
LGTM, had various non-blocking comments / questions. In particular I think this could stand to have more context/positioning w.r.t. where it lives vs MOLT Fetch since it seems to express overlapping functionality (but obv fills gaps on features that Fetch doesn't have). It would be good to add more about that in future so the reader can quickly assess "do i need to read all this? or should i just use MOLT Fetch?"
|
|
||
| This cookbook provides ready-to-use examples that demonstrate real-world uses of the [userscript API]({% link molt/userscript-api.md %}). You can copy and paste them into your own code, and you can adapt them for your specific use cases. | ||
|
|
||
| [MOLT Fetch]({% link molt/molt-fetch.md %}) does **not** support userscripts. Below each example, you will see the equivalent way of carrying out that transformation using MOLT Fetch, if it's possible to do so. When performing an [initial data load followed by live replication]({% link molt/migrate-load-replicate.md %}), **apply equivalent transformations in both the Fetch command and Replicator userscript** to ensure data consistency. |
There was a problem hiding this comment.
huh ? as someone unfamiliar with this product area i'm like "oh ... ok then"
so ... is this replacing that? or is it "yet another thing"? since you mentioned possible equivalent workflows that raises the q
maybe it's said elsewhere but IMO it would be good to add more context to this intro about "why should i use this instead of MOLT Fetch?" there isn't much product positioning here beyond the "you can do custom transforms" which yes, but when and why? probably not for this PR since it's due ~today but I think this is important
There was a problem hiding this comment.
For more context here, there are two different phases, MOLT Fetch for the initial bulk data load (which uses transformation json files), and MOLT Replicator for live replication (which uses userscripts). Users may have tailored needs for their migrations which may require the usage of MOLT Fetch transformations and MOLT Replicator userscripts. MOLT Replicator userscripts will typically complement MOLT Fetch transformations if those are used.
The cookbook examples provide more insight into common scenarios that the user may need to address, like if the source database table names and target database table names differ. MOLT Fetch transformations can be used to account for this naming difference during the initial bulk data load, and MOLT Replicator userscripts can be used during live replication afterwards to ensure that data rows are replicated to the correct target database table name from the original source database table name.
Additionally, live replication behavior can be heavily customized with userscripts, like creating a new audit table on the target database and storing specific data row changes there, implementing a dead-letter queue so that any data rows that fail to replicate can be sent to a dead-letter queue table on the target, and more use cases.
| }); | ||
| ~~~ | ||
|
|
||
| #### MOLT Fetch equivalent |
There was a problem hiding this comment.
again, it would be good to know WHY we are doing this compare/contrast - is Fetch deprecated? (maybe i should look at the Fetch docs, but as a user this area is fresh to me, and this is my immediate question)
There was a problem hiding this comment.
Fair, these are included because if the user initially uses a transformation with MOLT Fetch, the subsequent MOLT Replicator execution must do the equivalent transformation as well with a userscript to follow suit. Perhaps this context could be beneficial to include somewhere, if not already?
There was a problem hiding this comment.
@noelcrl This leads to a product-level question I have: if Fetch transformations provide less functionality than Replicator userscripts, but both need to be used in tandem (i.e. the transformations and userscripts need to be equivalent) when performing a bulk load + live replication, then what use is any of the userscript functionality that Fetch can't do? For example, Fetch can't create computed columns (as mentioned in the Compute New Columns cookbook example). I guess to take advantage of this functionality, then, you'd need to do a bulk load, then write some other script that computes the columns for that newly-loaded data, in place on the target, and then you would start live replication. Something like that?
There was a problem hiding this comment.
Yes, we would typically do the manual fix-ups after bulk load with MOLT Fetch, then have user scripts take care of the remaining live replication changes during MOLT Replicator, until a transformation receives support for MOLT Fetch.
noelcrl
left a comment
noelcrl
left a comment
There was a problem hiding this comment.
LGTM, thanks for the updates and clarifications!
…molt-user-scripts
5 participants
DOC-15772
Myself and @taroface are adding the userscript documentation to the existing MOLT docs.