borg on haiku

[ThomasWaldmann]

I was curious whether borg could work on haiku, so I tried borg 1.1.7 on Haiku OS R1/beta1 32bit:

borg seems to work.

I am not a haiku user, so I won't do the necessary changes to borg nor more extensive testing (nor packaging for haiku), but PRs are welcome.

Here are some notes about it for whoever wants to work on it:

I started with installing Haiku OS R1/beta1 from the ISO into VirtualBox - see their notes about installing into VirtualBox (esp. the hint about the Pro/1000 NIC card, to get a working network connection).

Then I installed some obviously needed stuff (see our docs): python3, pip3

Also installed zstd and lz4 (but guess this is not really needed as we bundle them).

I didn't install libacl nor libattr, haiku does not have them. Also no libb2 (blake2) in haiku, but borg bundles it, so no problem.

My first tries failed because it did not find the openssl headers (see code in our setup.py).

There are 2 problems:

• the openssl header location on haiku is not in our default search list
• even if it was, it would not work as it has headers/ instead of include/ in the path component after the prefix (which is hardcoded in current setup.py)

I hacked around them by creating a symbolic link in my home dir like include --> /boot/.../headers/ and then pointed BORG_OPENSSL_PREFIX to my home dir. :-)

After that, it stumbled over the too old default gcc 2.95. The setarch x86 command fixed that, then it compiled successfully with gcc 7.

I did some quick manual tests, borg init, create, list seemed to work ok.

TODO:

• FIXED by use pkg-config / refactor setup code #4437? : fix setup.py to be more flexible finding header files
• run the automated test suite of borg on haiku
• do more practical testing
• test on 32 and 64bit haiku
• if there is platform specific stuff a backup program needs to support for haiku / BeFS, implement it.

[mmuman]

Hi,
thanks for trying Haiku :-)
(yes, it's just "Haiku")

I didn't try Borg yet, but I've head quite good stuff about it.

I don't have much time now to try as I already have a long list of things to port, but I can give some hints.

Attributes… well, first, Haiku's concept of xattrs (inherited from BeOS) is a bit different than the (already numerous) variants is found in *nix. First, they aren't just name/value pairs, but triplets, since they also have a type (a 32bit 4CC). This allows many interesting things like indexing them properly, storing int and float values faster than having to parse strings, and knowing how to display them properly in the GUI (with more help from the MIME database). They aren't limited in size, as they can be allocated just like regular files, although usually very small (but when cross-building Haiku on Linux we did hit the ext2 limit of 2k/inode for some apps so we added wrappers around that). Also, we make extensive use of xattrs (MIME type, preferred app, even icons…). In fact, some files, like People files (contacts) are empty files with many xattrs. So for a backup software to actually be useful on Haiku it really must support them properly.

There are dir-like functions to list a file's xattrs (fs_open_attr_dir) and read them.

Then BFS itself has indices (B+Trees based hidden directories) for the attributes we ask it to maintain. the kernel has entry points to open index as dirs, and also run queries to find files matching formulas on attributes. I don't think indices really need backuping since they are just shadow view of the fs (except maybe the actual list of indices themselves).

You'll find more infos about attributes in Storage Kit in the BeBook (oddly we don't mention the attr functions in our API Guide), Programming guides, the book describing the BFS : Practical Filesystem Design.

We do have some libattr schim around, but it's not complete IIRC. At least I recall not finding the mangling scheme very correct. Btw, it's not a Haiku-only issue, NTFS implementations on Linux also differ in their mangling schemes and screw up. I wrote a paper about properly storing xattrs on various systems for DC2011, but it seems nobody reads papers. And sadly they are always left of things like NextCloud and other things. I'm not even sure our rsync port handles them correctly. I guess I'll have to fix that myself.

Now, we don't have any concept of ACL at the moment, and no port of libacl (which was the only reason for the xattrs to ever exist before they were withdrawn from POSIX IIRC). Historically BFS had a (not really used) SYS:* namespace reserved for system stuff (similar to the kernel.* on ext2), which we reused for packagefs, which has SYS:PACKAGE which returns which package a file comes from.

The rest of the fs should be visible through the POSIX API. Our stat struct has some more fields like create time, that BFS maintains.
BFS itself doesn't support hardlinks (because it stores the parent dir inode in the file inode to allow building the path back from dev/ino for queries).

Aside from live queries, we also support node monitoring (which BeOS had way before Linux ever had any of the inotify ancestors) with a different notification mechanism than signals though, but I don't know if it's used in Borg at all. I just wrote support for that in the owncloud client, I should publish that soon.

About gcc, we use gcc2 for binary compatibility, but unless for libs required at core level it's fine to build with newer gcc. It's handled almost like 32/64bit on linux: separate directories for libs (and sometimes headers). I usually just export PATH="/bin/x86:$PATH" but setarch x86 is fine.

Since we use headers instead of include using a common PREFIX dir for things usually never work correctly (and it's not a very good practice to assume this anyway). So it's better to be able to pass header and lib paths separately. You should be able to use pkg-config to locate openssl. Make sure you have openssl_devel installed (or openssl_x86_devel if using x86 arch so gcc > 2).

I didn't find any libb2 in HaikuPorts, but I suppose the builtin is ok.

Once you have something that builds, you'll want to try and write a HaikuPorter recipe.

Don't hesitate to come and ask on IRC.

[ThomasWaldmann]

@mmuman thanks for the detailled comment, guess it will be very helpful for whoever will do the port.

Adding some comments from #haiku on irc.freenode.net:

@mmu_man$ hmm I think we do have a libattr around, at least our own
@mmu_man$ since we make extensive use of xattrs, a backup that doesn't copy those is quite useless
@waddlesplash$ ThomasWaldmann: we have wrappers to the GNU xattr API but this isn't very useful because it doesn't get typed attributes
@waddlesplash$ so, you will need to use our native xattr API
@mmu_man$ ah yes
@mmu_man$ hmm sure ?
@mmu_man$ I though it wrapped the type in the name or something alike
@mmu_man$ anyway it's suboptimal
@waddlesplash$ oh, yeah it does
@waddlesplash$ it would be preferable to store the attributes properly
@mmu_man$ yeah, now the problem is how to convert them and back
@mmu_man$ but it's not like I didn't write something about it in 2011
@mmu_man$ oh, we don't index the C attr API in the API docs

[ThomasWaldmann]

About pkg-config see issue #1925 and PR #4437 (merged into master now).

[ThomasWaldmann]

borg 1.2.0a5 (alpha quality) released, with pkg-config support.
can be used for testing, including platform porting, packaging.

[ThomasWaldmann]

borg 1.2.0 was released recently (see comment above) and there is bleeding edge stuff with some new requirements in master (see also #6458).

[Begasus]

No python expert here but wanted to see if it actually build, got past the openssl issue but then I face (not sure what's missing or how to fix atm):

[x86] /Opslag/wip/borg-1.2.0> python3 setup.py build
Warning: can not import pkgconfig python package.
Detected OpenSSL [via BORG_OPENSSL_PREFIX]
Using bundled LZ4
Using bundled ZSTD
Using bundled xxhash
Traceback (most recent call last):
  File "setup.py", line 301, in <module>
    python_requires='>=3.8',
  File "/packages/python3_x86-3.7.12-1/.self/lib/python3.7/vendor-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/packages/python3_x86-3.7.12-1/.self/lib/python3.7/distutils/core.py", line 108, in setup
    _setup_distribution = dist = klass(attrs)
  File "/packages/python3_x86-3.7.12-1/.self/lib/python3.7/vendor-packages/setuptools/dist.py", line 456, in __init__
    k: v for k, v in attrs.items()
  File "/packages/python3_x86-3.7.12-1/.self/lib/python3.7/distutils/dist.py", line 292, in __init__
    self.finalize_options()
  File "/packages/python3_x86-3.7.12-1/.self/lib/python3.7/vendor-packages/setuptools/dist.py", line 807, in finalize_options
    ep(self)
  File "/packages/python3_x86-3.7.12-1/.self/lib/python3.7/vendor-packages/setuptools/dist.py", line 814, in _finalize_setup_keywords
    ep.load()(self, ep.name, value)
  File "/Opslag/wip/borg-1.2.0/.eggs/setuptools_scm-6.4.2-py3.7.egg/setuptools_scm/integration.py", line 75, in version_keyword
    _assign_version(dist, config)
  File "/Opslag/wip/borg-1.2.0/.eggs/setuptools_scm-6.4.2-py3.7.egg/setuptools_scm/integration.py", line 51, in _assign_version
    _version_missing(config)
  File "/Opslag/wip/borg-1.2.0/.eggs/setuptools_scm-6.4.2-py3.7.egg/setuptools_scm/__init__.py", line 107, in _version_missing
    f"setuptools-scm was unable to detect version for {config.absolute_root}.\n\n"
LookupError: setuptools-scm was unable to detect version for /Opslag/wip/borg-1.2.0.

Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj

[Begasus]

OK, after switching to python3.8 setting some export paths it seemed to work (don't know how it's used, but build seems ok) :)

EDIT also used a git checkout instead of the release tarball

[ThomasWaldmann]

@Begasus thanks for trying that out!

If you pip install pkgconfig before trying to install borg, you could check whether the pkg-config based discovery of the libs and headers work.

1.2-maint branch and master branch have quite different requirements, so it would be cool to try both. master "unbundled" a lot of stuff and requires some new libs.

[Begasus]

If you pip install pkgconfig before trying to install borg

@ThomasWaldmann pkg-config is always around (I do quit some of the ports at haikuports (with recently a new addition to libdeflate)), I also do a check with pkg-config afterwords to see if --cflags and --libs yield the correct vallues, some are not providing .pc file so yes they need the export value (with the pointer you gave in the first comment here it was easy to set them).

I'll be sure to try out 1.2 branch also, the one I did was from master.

[ThomasWaldmann]

You had pkg-config(the cli tool), but not pkgconfig (the python package):

[x86] /Opslag/wip/borg-1.2.0> python3 setup.py build
Warning: can not import pkgconfig python package.

[ThomasWaldmann]

But great that you did get master branch working! 1.2 should be easier even.

[mmuman]

Yay!
Next will be sorting out the xattr mess.

[Begasus]

You had pkg-config(the cli tool), but not pkgconfig (the python package):

Created a new recipe to build the python version for pkgconfig, when checking the master branch it detects, crypto, lz4, libdeflate and zstd, doesn't seem to find xxhash (although that has a .pc file included and points to the correct path.

[x86] /Opslag/wip/borg> python3.8 setup.py build
Detected and preferring libcrypto [via pkg-config]
Detected and preferring liblz4 [via pkg-config]
Detected and preferring libzstd [via pkg-config]
Detected and preferring libxxhash [via BORG_LIBXXHASH_PREFIX]
Detected and preferring libdeflate [via pkg-config]

[Begasus]

Yay! Next will be sorting out the xattr mess.

I'll leave that part for you! :P

[ThomasWaldmann]

@Begasus did you try letting it find xxhash without having BORG_LIBXXHASH_PREFIX set? if that is set, it will not try via pkgconfig. if so, check the .pc file (0.8.0 had bugs, iirc the version was not correctly set in there).

[Begasus]

@Begasus did you try letting it find xxhash without having BORG_LIBXXHASH_PREFIX set? if that is set, it will not try via pkgconfig. if so, check the .pc file (0.8.0 had bugs, iirc the version was not correctly set in there).

Oh nice catch, I did try without (failing), but we do have 0.8.0 in our ports installed, maybe time for an update :)
EDIT looking at the .pc file version is indeed empty

[Begasus]

Also no libb2 (blake2) in haiku

Fixed :) libb2 is now in haikuports, in the meantime also fixed our xxhash

Build time dependencies are OK, runtime dependencies still not yet (having a hard time with argon2-cffi) (PS cffi also bumped and is now available for the major python versions)

[ThomasWaldmann]

In case argon2-cffi becomes a bigger issue, you could also take borg 1.2.x (1.2-maint branch) for porting, which does not yet require that (1.2 also has quite some bundled code and differences in setup*.py though).

[Begasus]

No luck so far on argon2-cffi (did one for python-build that's probably needed for that, but no luck so far)
argon2-cffi-bindings should be good (first build check in Terminal is OK)
OK, list of packages on Haiku added/updated:

1. xxhash
2. libb2
3. pkgconfig
4. cffi
5. argon2-cffi-bindings

Still leaves argon2-cffi (just can't understand that build system)

EDIT 2 pulled in the latest sources for master and 1.2-maint, both install fine from Terminal, dependencies list for 1.2-maint is indeed shorter and doesn't require argon2 module, for master it is the only one that I haven't been able to build so it pulls and installs it. For both I don't have to use export ... anymore too, all dependencies are found. Progress! ;)

[ThomasWaldmann]

Here are some additions to our Vagrantfile (master branch).

It does not really work yet, issues:

• does not find a lot of stuff (header files, commands, ...)
• does not rsync correctly after booting the VM
• does not execute the packages_haiku script

If somebody who knows Haiku better could make this working, we'ld have a permanent testing platform for release testing:

+def packages_haiku
+  return <<-EOF
+    pkgman refresh
+    # check the pkgs again, maybe not all are needed
+    pkgman install -y python39 setuptools_python39 wheel_python39 pip_python39 libffi_devel cffi_python39
+    pkgman install -y cython_python39 packaging_python39 setuptools_scm_python39 pytest_python39 python_pkgconfig_python39
+    pkgman install -y argon2_cffi_bindings_python39 msgpack_python39
+    pkgman install -y attr_devel lz4_devel zstd_devel libb2_devel xxhash_devel libdeflate_devel
+  EOF
+end
 
+  config.vm.define "haiku64" do |b|
+    b.vm.box = "haiku-os/r1beta3-x86_64"
+    b.vm.provider :virtualbox do |v|
+      v.memory = 2048 + $wmem
+    end
+    b.vm.provision "fs init", :type => :shell, :inline => fs_init("vagrant")
+    b.vm.provision "packages haiku", :type => :shell, :inline => packages_haiku
+    b.vm.provision "build env", :type => :shell, :privileged => false, :inline => build_sys_venv("haiku")
+    b.vm.provision "install borg", :type => :shell, :privileged => false, :inline => install_borg("nofuse")
+    b.vm.provision "run tests", :type => :shell, :privileged => false, :inline => run_tests("haiku64", ".*fuse.*")
+  end

[Begasus]

Never used vagrant but checked (after install on Ubuntu where I run my VBox on Haiku), used the above suggesting in vagrantfile.
After boot it fails authenticating: haiku64: Warning: Authentication failure. Retrying... (keeps in a loop), after a while it times out.

[ThomasWaldmann]

@Begasus auth just worked for me (i have vagrant 2.2.9 with virtualbox, in case it makes a difference). it is normal that it does some retries while it works for the box to boot and sshd become ready, but in the end it should work.

[Begasus]

@ThomasWaldmann if I use vagrant ssh haiku64 it seems to be ok (I get into the Haiku shell), running vagrant up haiku64 still fails to connect ... (vagrant 2.2.6 it seems here)
EDIT connects with 2.2.19

[Begasus]

I don't know how rsync works in this, one thing I could think of is that every thing related to system directories is read-only, tried changing the mkdir path to use /boot/vagrant/borg/borg, but it still fails with: No such file or directory ...

[ThomasWaldmann]

After running against these rsync issues, I just used git clone https://... to get the code into the VM. Not really a solution, but unblocks one to work on the other issues in parallel. But then I also ran into git issues, which behaved strangely when I was working in /vagrant/borg/ directory - in the home dir, it worked better.

[Begasus]

Will write some stuff I encounter here to sum up what's been going on:
Changed to path for rsync to use /boot/home/borg instead of /vagrant/borg/borg (this syncs the directories, but don't seem to sync the files?)
Then running vagrant up haiku64 errors out with:

==> haiku64: Rsyncing folder: /home/begasus/borg/ => /boot/home/borg
There was an error when attempting to rsync a synced folder.
Please inspect the error message below for more info.

Host path: /home/begasus/borg/
Guest path: /boot/home/borg
Command: "rsync" "--verbose" "--archive" "--delete" "--exclude" ".python-version" "--no-owner" "--no-group" "--rsync-path" "rsync -zz" "-e" "ssh -p 2222 -o LogLevel=FATAL   -o ControlMaster=auto -o ControlPath=/tmp/vagrant-rsync-20220411-17963-1evyxum -o ControlPersist=10m  -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i '/home/begasus/borg/.vagrant/machines/haiku64/virtualbox/private_key'" "--exclude" ".vagrant/" "/home/begasus/borg/" "[email protected]:/boot/home/borg"
Error: rsync: [sender] write error: Broken pipe (32)
rsync error: error in rsync protocol data stream (code 12) at io.c(829) [sender=3.1.3]

Shut down the virtual machine with vagrant halt haiku64 and launched it again, inside the virtual machine I deleted the /boot/home/borg directory and ran git clone ... to have the files in /boot/home/borg, relaunched and rsync seems to run ok for a while but then errors out with:

==> haiku64: Rsyncing folder: /home/begasus/borg/ => /boot/home/borg
There was an error when attempting to rsync a synced folder.
Please inspect the error message below for more info.

Host path: /home/begasus/borg/
Guest path: /boot/home/borg
Command: "rsync" "--verbose" "--archive" "--delete" "--exclude" ".python-version" "--no-owner" "--no-group" "--rsync-path" "rsync -zz" "-e" "ssh -p 2222 -o LogLevel=FATAL   -o ControlMaster=auto -o ControlPath=/tmp/vagrant-rsync-20220411-23278-178qsjq -o ControlPersist=10m  -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i '/home/begasus/borg/.vagrant/machines/haiku64/virtualbox/private_key'" "--exclude" ".vagrant/" "/home/begasus/borg/" "[email protected]:/boot/home/borg"
Error: ERROR: out of memory in map_ptr [receiver]
rsync error: error allocating core memory buffers (code 22) at util2.c(105) [receiver=3.1.3]

[ThomasWaldmann]

Looks like rsync on haiku is broken / buggy?

[Begasus]

Running a search on Google shows more OS's that dealt with this? Both Ubuntu and Haiku here are running 3.1.3, from what I read so far it should be fixed in a 3.2.* version?

[ThomasWaldmann]

Hmm, I use vagrant a lot (see Vagrantfile) and usually rsync works. Sometimes vagrant had issues trying to install rsync if it is not already in the VM, but I usually avoid that just by having it in the VM beforehands.

[Begasus]

From the look of it it installs it when I launch haiku64 the first time:

==> haiku64: Installing rsync to the VM...

[Begasus]

PR for borg on Haiku up now: haikuports/haikuports#6921 👍

[Begasus]

Did some checks as shown in the first movie, looks to be working fine 👍

[ThomasWaldmann]

OK, so I guess we can close this after the PR is merged?

In case of issues or feature requests (which I guess are rather likely), we can just open new tickets.

[Begasus]

I thinks it's ok to close this one and the other one #6510
Even fidled a bit with borgweb, got a connection, but not sure if I need to do something more, anyway, first runs seem to be OK :)

[ThomasWaldmann]

borgweb is a rather specialized application, not very general purpose. and it didn't get much love in recent years.

way better for desktop users / GUI users are vorta and pika-backup, but i have no idea whether they could be ported to haiku.

for people who don't want a GUI, but want to configure their backups in a config file, there is also borgmatic as a quite popular add-on.

[ThomasWaldmann]

All done, thanks to @Begasus for porting / packaging!

[ThomasWaldmann]

Guess the port will need some testing for upcoming borg 1.4 (see 1.4-maint branch, I'll also release a new beta soon), e.g. rather using pip than calling setup.py.

[Begasus]

Thanks on the reminder, I'll do some testing here. +1

[Begasus]

Still on par, bumped our version at haikuports to the latest release, checked 1.4-maint branch with vorta 0.8.12 (we don't have qt6webengine yet). :)

[ThomasWaldmann]

@Begasus Thanks for checking!