Borg performance on an old CPU

[Aztorius]

Hi,
I use borg 1.4 on old servers (Intel(R) Xeon(R) CPU E5-2407 0 @ 2.20GHz) to backup data to a brand new external server (AMD Ryzen 7 PRO 3700 8-Core Processor). We have a 1Gigabit/s link between those. I want to backup large files (multiple gigabytes per file) for a total of tenths of TB of data. This data is on a ZFS pool of 6 disks (3 pairs of mirrors). Raw read of huge files is around 230MB/s with dd.
borg create is really (really) slow, around 15-20MB/s. I need it to go to 110MB/s (network link) to at least do the backup in one or two weeks instead of one or two months.
Also note that my CPU is not going to 100%, nor my storage. So I am wondering what is the root cause of the slow speed.

I read a lot of the doc from borg and i have changed these settings :

• encryption is using blake2 (a bit faster than sha2 as this is not accelerated by my CPU)
• compression lz4, but it is already the default
• chunker-params fixed,4194304 (and tried to increase this) but no real effect
• noacls to speedup (a bit) the backup with small files as this is a samba server

Here is the lscpu from client :

root@sinistra:~# lscpu
Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          46 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   4
  On-line CPU(s) list:    0-3
Vendor ID:                GenuineIntel
  Model name:             Intel(R) Xeon(R) CPU E5-2407 0 @ 2.20GHz
    CPU family:           6
    Model:                45
    Thread(s) per core:   1
    Core(s) per socket:   4
    Socket(s):            1
    Stepping:             7
    CPU(s) scaling MHz:   55%
    CPU max MHz:          2200,0000
    CPU min MHz:          1200,0000
    BogoMIPS:             4400,12
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi m
                          mx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs 
                          bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_c
                          pl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_
                          timer aes xsave avx lahf_lm pti tpr_shadow flexpriority ept vpid xsaveopt dtherm arat pln 
                          pts vnmi
Virtualization features:  
  Virtualization:         VT-x
Caches (sum of all):      
  L1d:                    128 KiB (4 instances)
  L1i:                    128 KiB (4 instances)
  L2:                     1 MiB (4 instances)
  L3:                     10 MiB (1 instance)
NUMA:                     
  NUMA node(s):           1
  NUMA node0 CPU(s):      0-3
Vulnerabilities:          
  Gather data sampling:   Not affected
  Itlb multihit:          KVM: Mitigation: VMX disabled
  L1tf:                   Mitigation; PTE Inversion; VMX conditional cache flushes, SMT disabled
  Mds:                    Vulnerable: Clear CPU buffers attempted, no microcode; SMT disabled
  Meltdown:               Mitigation; PTI
  Mmio stale data:        Unknown: No mitigations
  Reg file data sampling: Not affected
  Retbleed:               Not affected
  Spec rstack overflow:   Not affected
  Spec store bypass:      Vulnerable
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Mitigation; Retpolines, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
  Srbds:                  Not affected
  Tsx async abort:        Not affected

Here is the crud benchmark with borg1.4 done with the repo on local SSD and benchmark data on ZFS (repo is configured with blake2 encryption) :

root@sinistra:/share# BORG_PASSPHRASE="test" borg benchmark crud /root/borg_repotest ./borg_testdata/
C-Z-BIG         311.09 MB/s (10 * 100.00 MB all-zero files: 3.21s)
R-Z-BIG          86.11 MB/s (10 * 100.00 MB all-zero files: 11.61s)
U-Z-BIG         956.51 MB/s (10 * 100.00 MB all-zero files: 1.05s)
D-Z-BIG        3017.01 MB/s (10 * 100.00 MB all-zero files: 0.33s)
C-R-BIG          24.37 MB/s (10 * 100.00 MB random files: 41.04s)  <== This is the line i'm most interested in
R-R-BIG          68.18 MB/s (10 * 100.00 MB random files: 14.67s)
U-R-BIG         665.51 MB/s (10 * 100.00 MB random files: 1.50s)
D-R-BIG        1698.78 MB/s (10 * 100.00 MB random files: 0.59s)
C-Z-MEDIUM      400.70 MB/s (1000 * 1.00 MB all-zero files: 2.50s)
R-Z-MEDIUM      107.57 MB/s (1000 * 1.00 MB all-zero files: 9.30s)
U-Z-MEDIUM     1013.22 MB/s (1000 * 1.00 MB all-zero files: 0.99s)
D-Z-MEDIUM     2731.24 MB/s (1000 * 1.00 MB all-zero files: 0.37s)
C-R-MEDIUM       22.75 MB/s (1000 * 1.00 MB random files: 43.96s)
R-R-MEDIUM       68.68 MB/s (1000 * 1.00 MB random files: 14.56s)
U-R-MEDIUM     1044.00 MB/s (1000 * 1.00 MB random files: 0.96s)
D-R-MEDIUM      597.68 MB/s (1000 * 1.00 MB random files: 1.67s)
C-Z-SMALL        20.90 MB/s (10000 * 10.00 kB all-zero files: 4.79s)
R-Z-SMALL        51.27 MB/s (10000 * 10.00 kB all-zero files: 1.95s)
U-Z-SMALL        30.80 MB/s (10000 * 10.00 kB all-zero files: 3.25s)
D-Z-SMALL       216.31 MB/s (10000 * 10.00 kB all-zero files: 0.46s)
C-R-SMALL         5.61 MB/s (10000 * 10.00 kB random files: 17.82s)
R-R-SMALL        38.54 MB/s (10000 * 10.00 kB random files: 2.59s)
U-R-SMALL        28.72 MB/s (10000 * 10.00 kB random files: 3.48s)
D-R-SMALL       117.38 MB/s (10000 * 10.00 kB random files: 0.85s)

Another crud benchmark on the same SSD for the repo and test data :

root@sinistra:~# BORG_PASSPHRASE="test" borg benchmark crud ./borg_repotest ./borg_testdata/
C-Z-BIG         285.49 MB/s (10 * 100.00 MB all-zero files: 3.50s)
R-Z-BIG         107.67 MB/s (10 * 100.00 MB all-zero files: 9.29s)
U-Z-BIG         792.51 MB/s (10 * 100.00 MB all-zero files: 1.26s)
D-Z-BIG        3256.14 MB/s (10 * 100.00 MB all-zero files: 0.31s)
C-R-BIG          21.60 MB/s (10 * 100.00 MB random files: 46.30s) <== Mostly the same as before
R-R-BIG          60.05 MB/s (10 * 100.00 MB random files: 16.65s)
U-R-BIG         549.62 MB/s (10 * 100.00 MB random files: 1.82s)
D-R-BIG        2420.66 MB/s (10 * 100.00 MB random files: 0.41s)
C-Z-MEDIUM      364.39 MB/s (1000 * 1.00 MB all-zero files: 2.74s)
R-Z-MEDIUM       91.90 MB/s (1000 * 1.00 MB all-zero files: 10.88s)
U-Z-MEDIUM      690.99 MB/s (1000 * 1.00 MB all-zero files: 1.45s)
D-Z-MEDIUM     3244.46 MB/s (1000 * 1.00 MB all-zero files: 0.31s)
C-R-MEDIUM       20.99 MB/s (1000 * 1.00 MB random files: 47.65s)
R-R-MEDIUM       60.51 MB/s (1000 * 1.00 MB random files: 16.52s)
U-R-MEDIUM      667.25 MB/s (1000 * 1.00 MB random files: 1.50s)
D-R-MEDIUM      552.27 MB/s (1000 * 1.00 MB random files: 1.81s)
C-Z-SMALL        22.93 MB/s (10000 * 10.00 kB all-zero files: 4.36s)
R-Z-SMALL        47.77 MB/s (10000 * 10.00 kB all-zero files: 2.09s)
U-Z-SMALL        23.16 MB/s (10000 * 10.00 kB all-zero files: 4.32s)
D-Z-SMALL       185.56 MB/s (10000 * 10.00 kB all-zero files: 0.54s)
C-R-SMALL         8.70 MB/s (10000 * 10.00 kB random files: 11.50s)
R-R-SMALL        37.39 MB/s (10000 * 10.00 kB random files: 2.67s)
U-R-SMALL        18.95 MB/s (10000 * 10.00 kB random files: 5.28s)
D-R-SMALL       103.83 MB/s (10000 * 10.00 kB random files: 0.96s)

Another crud benchmark on the same SSD but without encryption (speed is a bit improved but far from what I am expecting) :

root@sinistra:~# borg benchmark crud ./borg_repotest ./borg_testdata/
C-Z-BIG         315.83 MB/s (10 * 100.00 MB all-zero files: 3.17s)
R-Z-BIG         144.78 MB/s (10 * 100.00 MB all-zero files: 6.91s)
U-Z-BIG        1782.63 MB/s (10 * 100.00 MB all-zero files: 0.56s)
D-Z-BIG        7152.45 MB/s (10 * 100.00 MB all-zero files: 0.14s)
C-R-BIG          25.69 MB/s (10 * 100.00 MB random files: 38.93s) <== A bit better without encryption, but nothing exceptionnal
R-R-BIG          74.15 MB/s (10 * 100.00 MB random files: 13.49s)
U-R-BIG         604.54 MB/s (10 * 100.00 MB random files: 1.65s)
D-R-BIG        3045.20 MB/s (10 * 100.00 MB random files: 0.33s)
C-Z-MEDIUM      436.90 MB/s (1000 * 1.00 MB all-zero files: 2.29s)
R-Z-MEDIUM      135.30 MB/s (1000 * 1.00 MB all-zero files: 7.39s)
U-Z-MEDIUM     1269.86 MB/s (1000 * 1.00 MB all-zero files: 0.79s)
D-Z-MEDIUM     4654.59 MB/s (1000 * 1.00 MB all-zero files: 0.21s)
C-R-MEDIUM       21.71 MB/s (1000 * 1.00 MB random files: 46.07s)
R-R-MEDIUM       75.56 MB/s (1000 * 1.00 MB random files: 13.23s)
U-R-MEDIUM     1094.75 MB/s (1000 * 1.00 MB random files: 0.91s)
D-R-MEDIUM     1866.80 MB/s (1000 * 1.00 MB random files: 0.54s)
C-Z-SMALL        20.44 MB/s (10000 * 10.00 kB all-zero files: 4.89s)
R-Z-SMALL        48.92 MB/s (10000 * 10.00 kB all-zero files: 2.04s)
U-Z-SMALL        22.55 MB/s (10000 * 10.00 kB all-zero files: 4.43s)
D-Z-SMALL       327.47 MB/s (10000 * 10.00 kB all-zero files: 0.31s)
C-R-SMALL        13.52 MB/s (10000 * 10.00 kB random files: 7.39s)
R-R-SMALL        45.31 MB/s (10000 * 10.00 kB random files: 2.21s)
U-R-SMALL        23.35 MB/s (10000 * 10.00 kB random files: 4.28s)
D-R-SMALL       145.03 MB/s (10000 * 10.00 kB random files: 0.69s)

Here is the cpu benchmark with borg2.0.0b13 :

root@sinistra:~# ./borg2.0.0b13 benchmark cpu
Chunkers =======================================================
buzhash,19,23,21,4095    1GB        4.163s
fixed,1048576            1GB        0.186s
Non-cryptographic checksums / hashes ===========================
xxh64                    1GB        0.196s
crc32 (zlib)             1GB        0.917s
Cryptographic hashes / MACs ====================================
hmac-sha256              1GB        5.784s
blake2b-256              1GB        3.360s
Encryption =====================================================
aes-256-ctr-hmac-sha256  1GB        6.329s
aes-256-ctr-blake2b      1GB        6.454s
aes-256-ocb              1GB        0.898s
chacha20-poly1305        1GB        2.078s
KDFs (slow is GOOD, use argon2!) ===============================
pbkdf2                   5          0.897s
argon2                   5          0.841s
Compression ====================================================
lz4          0.1GB      0.046s
zstd,1       0.1GB      0.087s
zstd,3       0.1GB      0.105s
zstd,5       0.1GB      1.664s
zstd,10      0.1GB      3.272s
zstd,16      0.1GB      17.647s
zstd,22      0.1GB      20.300s
zlib,0       0.1GB      0.104s
zlib,6       0.1GB      3.923s
zlib,9       0.1GB      3.921s
lzma,0       0.1GB      30.436s
lzma,6       0.1GB      50.028s
lzma,9       0.1GB      41.566s
msgpack ========================================================
msgpack      100k Items 0.428s

Based on everything above :

• This is not a link/internet issue as I have low speed even with only local disks
• This is not a storage issue as I can read much faster than what is done by Borg, and I tested also on SSDs which brings the same speed as hard drives
• This is not a compression issue : lz4 is running fast anyway
• Encryption brings a bit of overhead but is not the root cause.
• I also tested changing the chunker which helps a bit but like encryption, its marginal

Questions :

• What can I do to achieve better speed and ultimately my 110MB/s ?
• Also, no checkpoint archive has been created which is quite weird because it should create one each 30 mins ?! How checkpoint-interval measure time ?

Thank you

[ThomasWaldmann]

Nothing special here, the initial backup has a high volume and thus is slow.

With borg 1.x, you won't be able to backup at network link speed. This is because borg 1.x is single-threaded and synchronous. There is a plan to do multithreading and internal queueing some day, but likely not in the near future. borg2 will have some improvements, but due to project scope, multithreading is for a future milestone after borg2.

What you can do is:

• partition your input data into smaller subsets
• back up the subsets to different repositories (you can do that in parallel)
• wait until the initial backup is completed, restart backups that got interrupted (they will make progress)
• after initial backup, it will be much faster (depends on daily changed amount of data).

If you want to see checkpoint archives, use borg list --consider-checkpoints ....

[Aztorius]

Thanks, I will use multiple subsets. Hopefully it will be enough. I guess that if I had a better CPU I can have better performance ?

[ThomasWaldmann]

The cpu bound operations would be faster than (e.g. the cdc chunker, the hashing, compression and encryption). It won't help with the delays that are caused by disk and network bandwidth and latency.

[ThomasWaldmann]

When doing the 2nd+ backup, also use borg create --list ... and check that all unchanged files have U status (added files A, modified files M).

If that is the case, the files cache is working ok and speed is fast. If you see a lot of unexpected M, something is wrong and you need to check the FAQ about --files-cache to improve the speed.

[Aztorius]

After a lot (a lot) of testing. It appears that my network connection between the remote server and me is much lower than what i expected (200Mb/s instead of 1Gb/s). I am testing everything to find the root cause. The server has 10Gb/s download and 1Gb/s upload and seems pretty stable. I have some doubts about my ISP real upload speed although it is a professional fiber advertised as 7Gb/s download and 1Gb/s upload. So, it doesn't seem to be a Borg issue at all.

[ThomasWaldmann]

Well, I guess it'll get a faster after fixing that, but initial backup for that huge amount of data will still take a long time. Try to parallelize process and repo wise.

[ThomasWaldmann]

I just noticed that this is with borg 2.0.0b13. For this, the kind of repository is also relevant:

• file: should be quite fast
• ssh: should be somehow fast
• sftp: has known slowness issues due to the protocol and no remote "borg serve" agent.
• rclone: depends very much on the kind of remote

In general: this is a beta, so don't use it for production backups.

[Aztorius]

I used borg2.0.0b13 just to launch "benchmark cpu". It helps to notice anything weird related to algorithm acceleration on CPU.