feat: implement hybrid cache architecture #1246

fabiovincenzi · 2025-10-13T11:28:03Z

Summary

Two-tier cache system for faster git operations: persistent bare repos + ephemeral working copies.

Architecture

.remote/cache/ - Shared bare repositories (persistent)
.remote/work/<push-id>/ - Per-push working copies (temporary)

Configuration

Add to proxy.config.json:

{
  "cache": {
    "maxSizeGB": 2,
    "maxRepositories": 50,
    "cacheDir": "./.remote/cache"
  }
}

Security

Each push gets unique working copy (.remote/work//)
Bare cache contains only git objects (no user files, no credentials)
sanitizeRepositoryConfig() immediately removes credentials from git config
Working copies deleted after push completes

netlify · 2025-10-13T11:28:23Z

✅ Deploy Preview for endearing-brigadeiros-63f9d0 canceled.

Name	Link
🔨 Latest commit	`e67ff36`
🔍 Latest deploy log	https://app.netlify.com/projects/endearing-brigadeiros-63f9d0/deploys/690e128af32d60000894a9b7

codecov · 2025-10-13T11:32:06Z

Codecov Report

❌ Patch coverage is 82.19178% with 39 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.27%. Comparing base (df6406b) to head (e67ff36).

Files with missing lines	Patch %	Lines
src/proxy/processors/push-action/git-operations.ts	66.66%	16 Missing and 3 partials ⚠️
src/proxy/processors/push-action/cache-manager.ts	85.89%	8 Missing and 3 partials ⚠️
src/proxy/processors/push-action/pullRemote.ts	84.31%	8 Missing ⚠️
src/proxy/processors/push-action/clearBareClone.ts	92.30%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1246      +/-   ##
==========================================
- Coverage   83.36%   83.27%   -0.10%     
==========================================
  Files          70       73       +3     
  Lines        3005     3204     +199     
  Branches      499      545      +46     
==========================================
+ Hits         2505     2668     +163     
- Misses        397      427      +30     
- Partials      103      109       +6

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: Fabio Vincenzi <[email protected]>

…zi/git-proxy into feature/hybrid-cache

06kellyjac

I'm looking forward to faster pulls!

Caches are a common place for logic or security problems so we may want added scrutiny and testing for this PR to be extra safe :)

Added some thoughts, not ran this locally just yet.

06kellyjac · 2025-10-24T13:46:56Z

src/proxy/processors/push-action/clearBareClone.ts

+  if (process.env.NODE_ENV === 'test') {
+    // TEST: Full cleanup (bare cache + all working copies)
+    try {
+      if (fs.existsSync('./.remote')) {
+        fs.rmSync('./.remote', { recursive: true, force: true });
+        step.log('Test environment: Full .remote directory cleaned');
+      } else {
+        step.log('Test environment: .remote directory already clean');
+      }
+    } catch (err) {
+      step.log(`Warning: Could not clean .remote directory: ${err}`);
+    }
+  } else {


I feel like a safer option that ensures what we're testing more closely matches prod would be to either do some separate cleanup as necessary in the tests, or adjust config to have a much lower max (or even 0) then we can see if CacheManager itself is doing the cleanup.

src/proxy/processors/push-action/git-operations.ts

src/proxy/processors/push-action/pullRemote.ts

src/config/index.ts

06kellyjac · 2025-10-24T14:01:07Z

src/proxy/processors/push-action/cache-manager.ts

+    // Sort repositories by last accessed (oldest first for removal)
+    const reposToEvaluate = [...stats.repositories].sort(
+      (a, b) => a.lastAccessed.getTime() - b.lastAccessed.getTime(),
+    );


I can't remember if we can use toSorted yet. If we still cant a comment to use toSorted in the future would help to find this later :)

I think it should be available from node 20 so yes I'll use it

src/proxy/processors/push-action/cache-manager.ts

06kellyjac · 2025-10-24T14:12:33Z

src/proxy/processors/push-action/cache-manager.ts

+      return 0;
+    }
+
+    return Math.round(totalBytes / (1024 * 1024)); // Convert to MB


Suggested change

return Math.round(totalBytes / (1024 * 1024)); // Convert to MB

return Math.ceil(totalBytes / (1024 * 1024)); // Convert to MB

rounding down would hide some amount of KB up to 511KB. Not the end of the world but safest to always round up rather than be shortchanged.

There's no conversion anymore since we are using bytes everywhere

jescalada

Just a code review for now - I'll be testing the code later!

I was wondering if you could do some profiling to verify/document the speed improvements with this PR. Would be nice to try it out with massive repos (for example https://github.com/backstage/backstage which is >10GB) to see the difference.

I'll try to do some profiling soon as well 🙂

src/proxy/processors/push-action/cache-manager.ts

src/proxy/processors/push-action/pullRemote.ts

06kellyjac · 2025-10-27T11:50:35Z

why on earth is backstage's repo 10GB? 😭
Even nixpkgs is only 6.1GB ATM on my machine

jescalada · 2025-10-27T12:50:24Z

@06kellyjac It might be because of the tons of plugins that come with the repo (many of which have redundant components). Pulling with git clone --filter=blob:none https://github.com/backstage/backstage.git seems to make things a lot more speedy, however I'm not sure if we want to do things that way in the pullRemote. Perhaps a config option to filter out the BLOBs could help with these massive repos?

…Loader.cacheDir

…zi/git-proxy into pr/fabiovincenzi/1246

jescalada

Did some manual testing and things seem to work great! When using the profiling script though, I had some issues with git clone hanging without any error messages on both the backend and the script execution.

I tried testing with Backstage which is massive and so hiccups like that are expected. However, we do want to make sure that even these large repos can get cloned correctly without fail, or at least give descriptive errors on fail.

jescalada · 2025-11-02T07:41:23Z

src/proxy/processors/push-action/README.md

+
+| Metric              | Without Cache (main) | With Cache (PR) | Improvement          |
+| ------------------- | -------------------- | --------------- | -------------------- |
+| **Cold Push**       | 20.63s               | 17.58s          | 15% faster           |


I think the Improvement column calculations might be incorrect 🤔 A push that took 20s and is now taking 6.68s is 20/6.68 = 2.99x faster.

I think it feels more impactful and easier to understand when formatting it as a multiplier (3x faster, etc.) rather than a percentage.

You're right! The percentages were showing time reduction (66% of time saved) rather than the speedup multiplier. Updated to show it as a multiplier, which is much clearer and more impactful. Thanks!

jescalada · 2025-11-02T07:52:27Z

benchmark-cache.sh

+
+# Configuration
+PROXY_URL="http://localhost:8000"
+GITHUB_REPO="${1:-fabiovincenzi/open-webui}"


Nitpick: We might want to add an explicit error message when executing benchmark-cache.sh but the chosen repo is not added to the authorised list.

Just edited to show original errors everywhere

jescalada · 2025-11-02T07:59:38Z

benchmark-cache.sh

+
+# Configuration
+PROXY_URL="http://localhost:8000"
+GITHUB_REPO="${1:-fabiovincenzi/open-webui}"


Perhaps we could also throw an error if the user didn't provide their own repo in the arguments rather than defaulting to your fork?

Oh, I tought i removed all the references of my github, I just realized I didn't commit those

jescalada · 2025-11-02T09:16:14Z

benchmark-cache.sh

+    echo ""
+
+    # Store results
+    echo "$push_number,$is_first,$CLONE_TIME,$PUSH_TIME,$TOTAL_TIME" >> results.csv


Maybe we should add the resulting .csvs to .gitignore? 🤔

jescalada · 2025-11-02T09:55:12Z

benchmark-cache.sh

+echo -e "${GREEN}✓ Cache cleared${NC}"
+echo ""
+
+measure_push() {


Wonder if we could improve error handling here to prevent the script from hanging? I tried pulling my backstage fork but it strangely stopped responding on the 3rd push:

I think now the error handling shoud be much better, I'm sorry I didn't push latest changes

jescalada · 2025-11-02T09:55:48Z

benchmark-cache.sh

+    START_CLONE=$(date +%s.%N)
+
+    rm -rf "$REPO_NAME" 2>/dev/null || true
+    git clone "$PROXY_REPO_URL" "$REPO_NAME" > clone.log 2>&1


Perhaps we could output the clone status just to make sure things are going well for larger repos too?

Suggested change

git clone "$PROXY_REPO_URL" "$REPO_NAME" > clone.log 2>&1

git clone "$PROXY_REPO_URL" "$REPO_NAME" 2>&1 | tee clone.log

fabiovincenzi added 4 commits October 10, 2025 16:18

feat: implement hybrid cache architecture for repository cloning

956f388

feat: add configurable cache limits via proxy.config.json

cc49057

test: update clearBareClone tests for hybrid cache structure

d6413ce

chore: fix metric logging

cf76665

github-actions bot added the feature label Oct 13, 2025

Merge branch 'main' into feature/hybrid-cache

e998d07

fabiovincenzi mentioned this pull request Oct 13, 2025

Improve push flexibility and pull times #1207

Open

Merge branch 'main' into feature/hybrid-cache

734621c

fabiovincenzi force-pushed the feature/hybrid-cache branch from 12b56bd to 734621c Compare October 20, 2025 07:42

fabiovincenzi and others added 7 commits October 20, 2025 09:44

Merge branch 'main' into feature/hybrid-cache

452eb18

Signed-off-by: Fabio Vincenzi <[email protected]>

feat: implement git-operations module with native git commands

168d9b0

refactor: update pullRemote to use git-operations module

08116ba

test: add comprehensive hybrid cache integration tests

992c862

chore: remove unused isomorphic-git dependency

d5e1b5b

Merge branch 'feature/hybrid-cache' of https://github.com/fabiovincen…

6d5f886

…zi/git-proxy into feature/hybrid-cache

Merge branch 'main' into feature/hybrid-cache

6bc0ddc

fabiovincenzi marked this pull request as ready for review October 22, 2025 14:13

fabiovincenzi requested review from 06kellyjac, jescalada and kriswest October 22, 2025 14:18

06kellyjac reviewed Oct 24, 2025

View reviewed changes

fabiovincenzi and others added 4 commits October 27, 2025 11:00

Merge branch 'main' into feature/hybrid-cache

627137b

chore: remove redundant cache config fallback

235e152

refactor: use bytes internally in CacheManager for consistency

c762b5e

chore: use toSorted()

ab28d78

jescalada reviewed Oct 27, 2025

View reviewed changes

fabiovincenzi and others added 14 commits October 27, 2025 14:38

refactor: remove test-only cleanup, rely on CacheManager limits

88bbce8

refactor: use spawnSync instead of execSync

1737bfd

test: increase timeout for git clone tests in ConfigLoader

d35d109

perf: use performance.now() instead of Date.now()

0a54773

refactor: use cache paths from configuration

133e5e6

feat: add mutex to prevent race conditions in cache operations

f03d686

perf: remove unnecessary sort from getCacheStats

bc0be9f

fix: add logging for silent errors in getDirectorySize

d07ed9c

refactor: rename cacheDir to repoCacheDir to disambiguate from Config…

2acaee7

…Loader.cacheDir

Merge branch 'main' into feature/hybrid-cache

49695ff

refactor: rename cacheDir to repoCacheDir to disambiguate from Config…

b5bc3d8

…Loader.cacheDir

Merge branch 'feature/hybrid-cache' of https://github.com/fabiovincen…

d110463

…zi/git-proxy into pr/fabiovincenzi/1246

docs: add readme and cache benchmark script

b073eb3

Merge branch 'main' into feature/hybrid-cache

928846d

fabiovincenzi requested review from 06kellyjac and jescalada October 31, 2025 14:48

jescalada reviewed Nov 2, 2025

View reviewed changes

fabiovincenzi and others added 5 commits November 5, 2025 11:53

chore: add results csv to gitignore

5b94ec9

chore: move and edit benchmark script

7c05bfb

refactor: use multiplier for speed improvement

a42bd2e

Merge branch 'main' into feature/hybrid-cache

fd23676

Merge branch 'main' into feature/hybrid-cache

bd14e59

jescalada mentioned this pull request Nov 6, 2025

feat: adds ssh support for git operations #987

Open

Merge branch 'main' into feature/hybrid-cache

e67ff36

fabiovincenzi requested a review from jescalada November 7, 2025 15:43

	return Math.round(totalBytes / (1024 * 1024)); // Convert to MB
	return Math.ceil(totalBytes / (1024 * 1024)); // Convert to MB

	git clone "$PROXY_REPO_URL" "$REPO_NAME" > clone.log 2>&1
	git clone "$PROXY_REPO_URL" "$REPO_NAME" 2>&1 \| tee clone.log

feat: implement hybrid cache architecture #1246

Are you sure you want to change the base?

feat: implement hybrid cache architecture #1246

Uh oh!

Conversation

fabiovincenzi commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Architecture

Configuration

Security

Uh oh!

netlify bot commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for endearing-brigadeiros-63f9d0 canceled.

Uh oh!

codecov bot commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

06kellyjac left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jescalada left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

06kellyjac commented Oct 27, 2025

Uh oh!

jescalada commented Oct 27, 2025

Uh oh!

jescalada left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

fabiovincenzi commented Oct 13, 2025 •

edited

Loading

netlify bot commented Oct 13, 2025 •

edited

Loading

codecov bot commented Oct 13, 2025 •

edited

Loading