borg on haiku

[ThomasWaldmann]

I was curious whether borg could work on haiku, so I tried borg 1.1.7 on Haiku OS R1/beta1 32bit:

borg seems to work.

I am not a haiku user, so I won't do the necessary changes to borg nor more extensive testing (nor packaging for haiku), but PRs are welcome.

Here are some notes about it for whoever wants to work on it:

I started with installing Haiku OS R1/beta1 from the ISO into VirtualBox - see their notes about installing into VirtualBox (esp. the hint about the Pro/1000 NIC card, to get a working network connection).

Then I installed some obviously needed stuff (see our docs): python3, pip3

Also installed zstd and lz4 (but guess this is not really needed as we bundle them).

I didn't install libacl nor libattr, haiku does not have them. Also no libb2 (blake2) in haiku, but borg bundles it, so no problem.

My first tries failed because it did not find the openssl headers (see code in our setup.py).

There are 2 problems:

• the openssl header location on haiku is not in our default search list
• even if it was, it would not work as it has headers/ instead of include/ in the path component after the prefix (which is hardcoded in current setup.py)

I hacked around them by creating a symbolic link in my home dir like include --> /boot/.../headers/ and then pointed BORG_OPENSSL_PREFIX to my home dir. :-)

After that, it stumbled over the too old default gcc 2.95. The setarch x86 command fixed that, then it compiled successfully with gcc 7.

I did some quick manual tests, borg init, create, list seemed to work ok.

TODO:

• FIXED by use pkg-config / refactor setup code #4437? : fix setup.py to be more flexible finding header files
• run the automated test suite of borg on haiku
• do more practical testing
• test on 32 and 64bit haiku
• if there is platform specific stuff a backup program needs to support for haiku / BeFS, implement it.

[mmuman]

Hi,
thanks for trying Haiku :-)
(yes, it's just "Haiku")

I didn't try Borg yet, but I've head quite good stuff about it.

I don't have much time now to try as I already have a long list of things to port, but I can give some hints.

Attributes… well, first, Haiku's concept of xattrs (inherited from BeOS) is a bit different than the (already numerous) variants is found in *nix. First, they aren't just name/value pairs, but triplets, since they also have a type (a 32bit 4CC). This allows many interesting things like indexing them properly, storing int and float values faster than having to parse strings, and knowing how to display them properly in the GUI (with more help from the MIME database). They aren't limited in size, as they can be allocated just like regular files, although usually very small (but when cross-building Haiku on Linux we did hit the ext2 limit of 2k/inode for some apps so we added wrappers around that). Also, we make extensive use of xattrs (MIME type, preferred app, even icons…). In fact, some files, like People files (contacts) are empty files with many xattrs. So for a backup software to actually be useful on Haiku it really must support them properly.

There are dir-like functions to list a file's xattrs (fs_open_attr_dir) and read them.

Then BFS itself has indices (B+Trees based hidden directories) for the attributes we ask it to maintain. the kernel has entry points to open index as dirs, and also run queries to find files matching formulas on attributes. I don't think indices really need backuping since they are just shadow view of the fs (except maybe the actual list of indices themselves).

You'll find more infos about attributes in Storage Kit in the BeBook (oddly we don't mention the attr functions in our API Guide), Programming guides, the book describing the BFS : Practical Filesystem Design.

We do have some libattr schim around, but it's not complete IIRC. At least I recall not finding the mangling scheme very correct. Btw, it's not a Haiku-only issue, NTFS implementations on Linux also differ in their mangling schemes and screw up. I wrote a paper about properly storing xattrs on various systems for DC2011, but it seems nobody reads papers. And sadly they are always left of things like NextCloud and other things. I'm not even sure our rsync port handles them correctly. I guess I'll have to fix that myself.

Now, we don't have any concept of ACL at the moment, and no port of libacl (which was the only reason for the xattrs to ever exist before they were withdrawn from POSIX IIRC). Historically BFS had a (not really used) SYS:* namespace reserved for system stuff (similar to the kernel.* on ext2), which we reused for packagefs, which has SYS:PACKAGE which returns which package a file comes from.

The rest of the fs should be visible through the POSIX API. Our stat struct has some more fields like create time, that BFS maintains.
BFS itself doesn't support hardlinks (because it stores the parent dir inode in the file inode to allow building the path back from dev/ino for queries).

Aside from live queries, we also support node monitoring (which BeOS had way before Linux ever had any of the inotify ancestors) with a different notification mechanism than signals though, but I don't know if it's used in Borg at all. I just wrote support for that in the owncloud client, I should publish that soon.

About gcc, we use gcc2 for binary compatibility, but unless for libs required at core level it's fine to build with newer gcc. It's handled almost like 32/64bit on linux: separate directories for libs (and sometimes headers). I usually just export PATH="/bin/x86:$PATH" but setarch x86 is fine.

Since we use headers instead of include using a common PREFIX dir for things usually never work correctly (and it's not a very good practice to assume this anyway). So it's better to be able to pass header and lib paths separately. You should be able to use pkg-config to locate openssl. Make sure you have openssl_devel installed (or openssl_x86_devel if using x86 arch so gcc > 2).

I didn't find any libb2 in HaikuPorts, but I suppose the builtin is ok.

Once you have something that builds, you'll want to try and write a HaikuPorter recipe.

Don't hesitate to come and ask on IRC.

[ThomasWaldmann]

@mmuman thanks for the detailled comment, guess it will be very helpful for whoever will do the port.

Adding some comments from #haiku on irc.freenode.net:

@mmu_man$ hmm I think we do have a libattr around, at least our own
@mmu_man$ since we make extensive use of xattrs, a backup that doesn't copy those is quite useless
@waddlesplash$ ThomasWaldmann: we have wrappers to the GNU xattr API but this isn't very useful because it doesn't get typed attributes
@waddlesplash$ so, you will need to use our native xattr API
@mmu_man$ ah yes
@mmu_man$ hmm sure ?
@mmu_man$ I though it wrapped the type in the name or something alike
@mmu_man$ anyway it's suboptimal
@waddlesplash$ oh, yeah it does
@waddlesplash$ it would be preferable to store the attributes properly
@mmu_man$ yeah, now the problem is how to convert them and back
@mmu_man$ but it's not like I didn't write something about it in 2011
@mmu_man$ oh, we don't index the C attr API in the API docs

[ThomasWaldmann]

About pkg-config see issue #1925 and PR #4437 (merged into master now).

[ThomasWaldmann]

borg 1.2.0a5 (alpha quality) released, with pkg-config support.
can be used for testing, including platform porting, packaging.

[ThomasWaldmann]

borg 1.2.0 was released recently (see comment above) and there is bleeding edge stuff with some new requirements in master (see also #6458).

[Begasus]

No python expert here but wanted to see if it actually build, got past the openssl issue but then I face (not sure what's missing or how to fix atm):

[x86] /Opslag/wip/borg-1.2.0> python3 setup.py build
Warning: can not import pkgconfig python package.
Detected OpenSSL [via BORG_OPENSSL_PREFIX]
Using bundled LZ4
Using bundled ZSTD
Using bundled xxhash
Traceback (most recent call last):
  File "setup.py", line 301, in <module>
    python_requires='>=3.8',
  File "/packages/python3_x86-3.7.12-1/.self/lib/python3.7/vendor-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/packages/python3_x86-3.7.12-1/.self/lib/python3.7/distutils/core.py", line 108, in setup
    _setup_distribution = dist = klass(attrs)
  File "/packages/python3_x86-3.7.12-1/.self/lib/python3.7/vendor-packages/setuptools/dist.py", line 456, in __init__
    k: v for k, v in attrs.items()
  File "/packages/python3_x86-3.7.12-1/.self/lib/python3.7/distutils/dist.py", line 292, in __init__
    self.finalize_options()
  File "/packages/python3_x86-3.7.12-1/.self/lib/python3.7/vendor-packages/setuptools/dist.py", line 807, in finalize_options
    ep(self)
  File "/packages/python3_x86-3.7.12-1/.self/lib/python3.7/vendor-packages/setuptools/dist.py", line 814, in _finalize_setup_keywords
    ep.load()(self, ep.name, value)
  File "/Opslag/wip/borg-1.2.0/.eggs/setuptools_scm-6.4.2-py3.7.egg/setuptools_scm/integration.py", line 75, in version_keyword
    _assign_version(dist, config)
  File "/Opslag/wip/borg-1.2.0/.eggs/setuptools_scm-6.4.2-py3.7.egg/setuptools_scm/integration.py", line 51, in _assign_version
    _version_missing(config)
  File "/Opslag/wip/borg-1.2.0/.eggs/setuptools_scm-6.4.2-py3.7.egg/setuptools_scm/__init__.py", line 107, in _version_missing
    f"setuptools-scm was unable to detect version for {config.absolute_root}.\n\n"
LookupError: setuptools-scm was unable to detect version for /Opslag/wip/borg-1.2.0.

Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj

[Begasus]

OK, after switching to python3.8 setting some export paths it seemed to work (don't know how it's used, but build seems ok) :)

EDIT also used a git checkout instead of the release tarball

[ThomasWaldmann]

@Begasus thanks for trying that out!

If you pip install pkgconfig before trying to install borg, you could check whether the pkg-config based discovery of the libs and headers work.

1.2-maint branch and master branch have quite different requirements, so it would be cool to try both. master "unbundled" a lot of stuff and requires some new libs.

[Begasus]

If you pip install pkgconfig before trying to install borg

@ThomasWaldmann pkg-config is always around (I do quit some of the ports at haikuports (with recently a new addition to libdeflate)), I also do a check with pkg-config afterwords to see if --cflags and --libs yield the correct vallues, some are not providing .pc file so yes they need the export value (with the pointer you gave in the first comment here it was easy to set them).

I'll be sure to try out 1.2 branch also, the one I did was from master.

[ThomasWaldmann]

You had pkg-config(the cli tool), but not pkgconfig (the python package):

[x86] /Opslag/wip/borg-1.2.0> python3 setup.py build
Warning: can not import pkgconfig python package.

[ThomasWaldmann]

But great that you did get master branch working! 1.2 should be easier even.