feat: archivistactl import - #358
Open
kairoaraujo wants to merge 2 commits into
Open
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #358 +/- ##
==========================================
- Coverage 82.40% 1.64% -80.76%
==========================================
Files 10 121 +111
Lines 358 28956 +28598
==========================================
+ Hits 295 477 +182
- Misses 43 28422 +28379
- Partials 20 57 +37 ☔ View full report in Codecov by Sentry. |
kairoaraujo
marked this pull request as draft
August 19, 2024 13:30
kairoaraujo
force-pushed
the
feat/import
branch
2 times, most recently
from
August 20, 2024 14:01
d8ff65a to
fe986e1
Compare
kairoaraujo
marked this pull request as ready for review
August 21, 2024 05:43
Introduces a new `archivistactl` import It allows Archivista users to import DSSE Envelopes directly to the Archivista database. The feature allows direct import, which can help importing huge amount of data as it use concurrency (go routines) to process. Signed-off-by: Kairo Araujo <kairo.araujo@testifysec.com>
Member
|
Definitely like this -- there's been a few times where I've hacked this in but never actually got it in a mergable state. I think one thing we may eventually want to do is also support smarter bulk loading on the server side. We can do some better SQL optimizations when we know we're going to load in a bunch of data, but we can worry about that later |
✅ Deploy Preview for archivista-dev ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
There was a problem hiding this comment.
Pull request overview
Adds a new archivistactl import subcommand intended to bulk-import DSSE envelopes directly into the Archivista SQL-backed metadata store, using a concurrency limit to improve throughput for large folders.
Changes:
- Introduces
archivistactl importcommand with--from-dir,--db-uri, and--max-concurrentflags. - Implements concurrent directory walking + per-file DSSE parsing and storage into the SQL store.
- Adds DB client initialization logic to connect to MySQL/Postgres via
sqlstore.NewEntClient.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+37
to
+43
| importCmd = &cobra.Command{ | ||
| Use: "import", | ||
| Short: "import dsses to the Archivista DB server", | ||
| SilenceUsage: true, | ||
| Args: cobra.MaximumNArgs(2), | ||
| RunE: importDsse, | ||
| } |
Comment on lines
+46
to
+53
| func init() { | ||
| rootCmd.AddCommand(importCmd) | ||
| importCmd.PersistentFlags().StringP("from-dir", "", "", "Directory to import from. Example: /path/to/directory") | ||
| importCmd.PersistentFlags().StringP("db-uri", "", "", "Database URI to import to. Supported schemes: mysql, psql. Example: mysql://user:password@localhost:3306/testify") | ||
| importCmd.PersistentFlags().IntP("max-concurrent", "", 3, "Maximum number of concurrent imports.") | ||
| err := importCmd.MarkPersistentFlagRequired("db-uri") | ||
| cobra.CheckErr(err) | ||
| } |
Comment on lines
+93
to
+102
| switch strings.ToUpper(purl.Scheme) { | ||
| case "MYSQL": | ||
| scheme = "MYSQL" | ||
| uri = purl.User.String() + "@" + purl.Host + purl.Path | ||
| case "PSQL": | ||
| scheme = "PSQL" | ||
| uri = dbURI | ||
| default: | ||
| return nil, fmt.Errorf("unsupported database scheme %s", purl.Scheme) | ||
| } |
Comment on lines
+179
to
+187
| ec, err := dbClient(dbURI) | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| sqlStore, _, err := sqlstore.New(context.Background(), ec) | ||
| if err != nil { | ||
| return err | ||
| } |
Comment on lines
+189
to
+195
| max, err := ccmd.Flags().GetInt("max-concurrent") | ||
| if err != nil { | ||
| fmt.Println("Failed to get max-concurrent flag", err) | ||
| } | ||
| fmt.Print("\nImporting DSSes from folder", sourceDir, " to the database server") | ||
| fmt.Print("\nMax concurrent imports: ", max, "\n\n") | ||
| importFile(sourceDir, sqlStore, max) |
Comment on lines
+117
to
+120
| func importFile(path string, sqlStore *sqlstore.Store, maxConcurrent int) { | ||
| fpaths, _ := walkDir(path) | ||
| var wg sync.WaitGroup | ||
| sem := make(chan struct{}, maxConcurrent) // Buffered channel acting as a semaphore |
Comment on lines
+141
to
+158
| if envelope.PayloadType != "" { | ||
| ngitoid, err := gitoid.New(bytes.NewReader(file), gitoid.WithContentLength(int64(len(file))), gitoid.WithSha256()) | ||
| if err != nil { | ||
| fmt.Println("Skipping file: "+fpath+" cannot generate valid GitOID", fpath) | ||
| return | ||
| } | ||
| err = sqlStore.Store(context.Background(), ngitoid.String(), file) | ||
| if err != nil { | ||
| // if failed due to duplicate entry, skip | ||
| if strings.Contains(err.Error(), "Duplicate entry") { | ||
| fmt.Println("Skipping file: " + fpath + " cannot store duplicated entry") | ||
| } else { | ||
| fmt.Println("Skipping file: "+fpath+" failed to import.", err) | ||
| } | ||
| return | ||
| } | ||
| } | ||
| fmt.Println("Successfully imported", fpath) |
Comment on lines
+46
to
+53
| func init() { | ||
| rootCmd.AddCommand(importCmd) | ||
| importCmd.PersistentFlags().StringP("from-dir", "", "", "Directory to import from. Example: /path/to/directory") | ||
| importCmd.PersistentFlags().StringP("db-uri", "", "", "Database URI to import to. Supported schemes: mysql, psql. Example: mysql://user:password@localhost:3306/testify") | ||
| importCmd.PersistentFlags().IntP("max-concurrent", "", 3, "Maximum number of concurrent imports.") | ||
| err := importCmd.MarkPersistentFlagRequired("db-uri") | ||
| cobra.CheckErr(err) | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it
Introduces a new
archivistactlimportIt allows Archivista users to import DSSE Envelopes directly to the Archivista database.
The feature allows direct import, which can help import huge amounts of data as it uses concurrency (go routines) to process.
Performance examples
Importing 2100 new DSSE Envelopes
Using default:
--max-concurrent 3Using
--max-concurrent 10Which issue(s) this PR fixes (optional)
(optional, using
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close the issue(s) when the PR gets merged)*Related #319
Acceptance Criteria Met
Special notes for your reviewer:
TODO:
--exist-firstfail in the first import error instead of skippingI can add the TODO as new commits or as new PRs.