Allow Gramps' core to be imported without requiring GObject

[bartfeenstra]

This is a first stab at making Gramps more easily usable (or at all) in a stand-alone fashion, without requiring third-party system packages to be installed manually. It also aims to be backwards-compatible, hence the addition of checks rather than a refactoring.

I tested this using https://github.com/bartfeenstra/betty/blob/gramps-api/test.py (which itself uses my own relatively extensive Gramps family tree export) and other than a warning for one of the plugins, this works fine and my code can successfully load the family tree:

ERROR: Failed reading plugin registration geography.gpr.py
Traceback (most recent call last):
  File "/home/bart/code/bartfeenstra/betty/venv13/lib/python3.13/site-packages/gramps/gen/plug/_pluginreg.py", line 1428, in scan_dir
    exec(
    ~~~~^
        compile(stream, filename, "exec"),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        make_environment(_=local_gettext),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        {"uistate": uistate},
        ^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "geography.gpr.py", line 33, in <module>
ModuleNotFoundError: No module named 'gi'

WARNING:._manager:Plugin error (from 'importprogen'): No module named 'gi'

[dsblank]

@bartfeenstra thanks for starting the process! @Nick-Hall, do you think that this would be a minor enough change to include in 6.0.1?

[PQYPLZXHGF]

Reference to related issues

• 0006537: Gramps starts the GUI even in command line mode
• 0006085: Import outside of the gen submodule
• 0005598: refactor GUI tools to use User and enable CLI operation

[Nick-Hall]

do you think that this would be a minor enough change to include in 6.0.1?

No. This should stay in the master branch.

[Nick-Hall]

When I wrote the code using GLib I also considered using platformdirs. At the time I didn't want to include another dependency, but it makes more sense now. We could use the same code for everything.

[bartfeenstra]

do you think that this would be a minor enough change to include in 6.0.1?

No. This should stay in the master branch.

As someone who's new to the development of Gramps itself, can I ask what it means for something to "stay in the master branch" in this project?

[emyoulation]

There are maintenance branches that have bugs fixes and translation enhancements.

The master branch is where enhancement pull requests and commits for the next major release are submitted.

[bartfeenstra]

And the next major release would be Gramps 7, or 6.1? Either way, looking at the Gramps release frequency, I am really hoping we can make some of these improvements in 6.0.1, 6.0.2, etc. in order for Betty to be able to start using Gramps APIs within a reasonable timeframe. The proposed changes are minor and aim not to break any existing functionality, but they would allow multiple third parties to integrate their software with Gramps. Could you perhaps explain why you are currently not inclined to consider these changes in the short term?

[Nick-Hall]

We only accept bug fixes in maintenance branches. This is clearly stated in our Committing policies.

[emyoulation]

And the next major release would be Gramps 7, or 6.1? Either way, looking at the Gramps release frequency, I am really hoping we can make some of these improvements in 6.0.1, 6.0.2, etc. in order for Betty to be able to start using Gramps APIs within a reasonable timeframe.

The next major release should be a 6.1.0 version.
The next maintenance release should be a 6.0.1 version.

2 things to consider...

First, you do not need to wait for a release of the GTK based Gramps for Desktops if you are going to base your Betty features on the Gramps engine that has been decoupled from the graphical interface. You can release a fork bundle on your own schedule.

Second, Gramps 6.0 introduced a lot of changes. And developers of advanced features typically want to see where the foundation settles before committing major effort. Now they can build on the 6.0 engine. And that is something that @DavidMStraub wheere seems interested for Gramps Web with its javascript-based GUI, and that @dsblank has expressed interest too. @cdhorn and @ztlxltl have designed new views that will also probably find some tweak that 6.0 needs to support their vision. If you and these developers coordinate new hooks for 3rd party expansion, then it would be a compelling argument for pushing up the release schedule of a 6.1 version.

(If the deferred Place GEPS 43 was was finalized, the early release argument would be even more compelling.)

[DavidMStraub]

This is a great initiative, it will also silence a warning in Gramps Web that has been annoying/confusing users for a long time.

Only one comment though:

GLib.get_user_data_dir() if _GOBJECT_AVAILABLE else USER_HOME, "gramps"
GLib.get_user_config_dir() if _GOBJECT_AVAILABLE else USER_HOME, "gramps"

I don't think it's good that we are changing the directories here. This means that, when a user installs gi, suddenly the directories change and trees are not found anymore. I think it would be better to check what those GLib methods do and to extract an equivalent functionality, perhaps also copy/pasting stomething from https://github.com/tox-dev/platformdirs (or even add it as a dependency now that we are slowly allowing pip depdencies)?

[jralls]

I don't think it's good that we are changing the directories here.

Indeed.

perhaps also copy/pasting something from https://github.com/tox-dev/platformdirs

That would be ill-advised since they appear not to know the correct paths on Windows:
"If on Windows (at least English Win) that should be:
"C:\Documents and Settings\<User>\Application Data\Local Settings\<AppAuthor>\<AppName>"

That was almost right on Windows 8 more than 15 years ago, but Windows 10 changed it to C:\Users\<user>\AppData\<Local|Roaming>\<App name>

I think it would be better to check what those GLib methods do and to extract an equivalent functionality,

No need to check, it was exhaustively discussed in #2013. They're trivial, effectively

def get_user_config_dir():
    if 'XDG_CONFIG_HOME' in os.environ:
        return os.environ['XDG_CONFIG_HOME']
    return os.path.join(os.environ['HOME'], ".config")

Substitute 'XDG_DATA_HOME' and os.path.join[os.environ['HOME'], ".local", "share") for user_data_dir.

[bartfeenstra]

This was a very rough first stab to explore the scope of the changes needed to make this happen. I agree the code changes as they exist now are not at all ready to be merged.

I've now got builds running that take the Betty changes to use Gramps' APIs using this PR's branch. So far I have managed to import everything Betty needs through Gramps' APIs without a hiccup. Thank you for your help so far! The current changes appear enough to be able to bootstrap Gramps without GObject and Cairo system dependencies. However, Gramps also directly runs gettext commands and expects those to be within the PATH.

Based on @DavidMStraub's and @jralls's suggestions I started looking through the glib C code to see if we could replicate the functionality in Python. So, the high-level functions are indeed as easy as @jralls showed us. But the fallbacks all call GLib.get_user_special_dir which is slightly more complex and calls out to yet several other functions. Then there is GLib.get_home_dir which poses a similar challenge. I'm not quite sure if we can tackle the original problem like this; we'd end up replicating a lot of glib C code in Python.

Having thought more about @DavidMStraub's concern of "what if gi suddenly becomes (un)available on a system, changing directies". I agree, but in which situations would that actually happen? Not in the desktop version, which comes packaged with everything Gramps needs. It would only be for those people who recently (since last year) started using Gramps as a pip dependency. Technically, still a BC break, so a valid concern. However, by narrowing this down, we can consider certain workarounds. What about a new global/env variable that can be set explicitly to enable the new, 'decoupled' behavior? Dependents would have to opt in, and nothing would change for existing users/dependents.

Next steps

• in this PR, try and find a long-term solution that is clean and does not introduce regressions
• in another PR, explore the gettext executable dependeny

[bartfeenstra]

GRAMPS_HOME seems to do a lot of what is needed already, except that for some directories it is considered to have priority and if it's set, glib won't be consulted. For others, such as the USER_PICTURES directory, glib has priority over any other methods to determine this directory.

So we'd need something like GRAMPS_HOME, but that, when set, is used as the base for any and all directories used by Gramps. Providing this variable would then simply prevent any of the other setup code from being run (and if we make glib imports local, we've solved the import-time dependency problem). For Betty specifically, this would also give us a way to run Gramps isolated within that directory, preventing it from finding directories elsewhere on the system on its own. That's not only good for testing, but if I understand Gramps correctly, this will also provide conflicts between Gramps the Betty dependency and Gramps the desktop application installed elsewhere on the system, that may well be different, incompatible versions, so they cannot use the same directories (databases).

[bartfeenstra]

Using a single, new environment variable (see the PR changes at the time of this post) I managed to load data from a file through Gramps into Betty, without needing to install additional OS packages, or running into import errors. I believe this is a very clean approach that does not change any existing behavior as perceptible by users or developers, and does not duplicate anything.

The current environment variable name is temporary. Suggestions for a permanent one? GRAMPSHOME_ISOLATED?

[jralls]

Why not instead test for GI being available and if not rely solely on GRAMPSHOME? I don't see the point of introducing a new environment variable.

[bartfeenstra]

Why not instead test for GI being available and if not rely solely on GRAMPSHOME? I don't see the point of introducing a new environment variable.

This PR originally proposed that, but moved away from that approach because of two requirements that must both be met:

1. @Nick-Hall correctly pointed out that we must not make any changes that will allow a Gramps installation's behavior to change when someone decides to install or uninstall glib at some point.
2. for proper library use, dependents such as Betty need full control over where Gramps puts its files. GRAMPS_HOME does not allow for this, because for USER_PICTURES GLib.get_user_special_dir() is called first, and only if that returns nothing it falls back onto USER_DATA which may be set by GRAMPSHOME.

Using a new variable means that

1. we do not break BC
2. this variable can provide the full isolation that GRAMPS_HOME does not

When development on Gramps 7 starts and BC breaks will be accepted, this new variable may be removed and its functionality merged into GRAMPS_HOME (or even more radical improvements may be made).

[jralls]

This PR originally proposed that, but moved away from that approach because of two requirements that must both be met:

@Nick-Hall correctly pointed out that we must not make any changes that will allow a Gramps installation's behavior to change when someone decides to install or uninstall glib at some point.

@Nick-Hall didn't say that, @DavidMStraub did. It's ridiculous on its face: The Gramps GUI won't work without GI, and GI depends on GLib,. It would have to be completely rewritten from scratch with a different GUI toolkit to make it so that it would. It is therefore not possible for the behavior of a Gramps installation to be unchanged by installing or uninstalling GLib.

for proper library use, dependents such as Betty need full control over where Gramps puts its files.

I don't think use as a library has ever been a Gramps design goal. If that's what you want Gramps for you should incorporate a fork of it into your project.

dependents such as Betty need full control over where Gramps puts its files.

If it's a library use then the dependent might validly want full control of where it puts its files, but it has no right to demand control over where Gramps puts its files. In that case the dependent app has to have its own file handling routines rather than using Gramps's. OTOH a dependent app could treat Gramps as a database server. Servers (think SQL database servers like MySQL) determine where the files are stored and hand off the data to clients.

Philosophical issues aside, the database and media paths in conf.py are defaults: The actual locations used can be overridden by preference settings. If you want "full control" you need to override those preference settings, not the fallbacks in const.py.

Config locations are best governed by the OS's conventions, which on Linux is the XDG Base Directory Spec, on macOS is ~/Library/Application Support, and on Windows is %APPDATA%.

GRAMPS_HOME does not allow for this, because for USER_PICTURES GLib.get_user_special_dir() is called first, and only if that returns nothing it falls back onto USER_DATA which may be set by GRAMPSHOME.

USER_PICTURES is solely used as a fallback media path if one isn't set in the database.

[bartfeenstra]

If it's a library use then the dependent might validly want full control of where it puts its files, but it has no right to demand control over where Gramps puts its files. In that case the dependent app has to have its own file handling routines rather than using Gramps's. OTOH a dependent app could treat Gramps as a database server. Servers (think SQL database servers like MySQL) determine where the files are stored and hand off the data to clients.

Actually, it's very common for libraries to allow dependencies (and I/O paths are dependencies) to be injected as the point of libraries is to be usable by other software. Without being able to determine these paths from calling code, using Gramps within a test run, for example, will impact live data on the system or be impacted by it, and that is a very big problem. In the same vein, this problem cannot be solved by using system settings, exactly because those cannot be isolated either. Interestingly, when writing unit or integration tests for any project, including Gramps, that project is treated like a library, because the tests only import and run the parts of the project they need, and must always be isolated from the wider system.

USER_PICTURES is solely used as a fallback media path if one isn't set in the database.

That still means that checking for glib would change the order of things compared to the current situation, and impact databases without media paths. Fine if The Powers That Be approve that, but until then it must be avoided.

[DavidMStraub]

@Nick-Hall perhaps it would be time to consider having a code of conduct (like we have for Gramps Web API) also for the Gramps main repo. @jralls quoting my statement as "ridiculous on its face" is not what I consider "Being respectful of differing opinions, viewpoints, and experiences".

I don't think use as a library has ever been a Gramps design goal. If that's what you want Gramps for you should incorporate a fork of it into your project.

It certainly hasn't been a design goal originally, but it's a fact that Gramps is being used as a library today, and not just by some niche private projects. You (may or) may not like Gramps Web, but it's an official community project living in the gramps-project namespace and has > 600 stars on Github - more than Webtrees or any other genealogy repository (except Gramps).

[jralls]

@DavidMStraub My sincere apologies. I didn't mean to insult you at all.

[emyoulation]

@jralls
As a person who has made many statements that were unintentionally ridiculous when considered within the confines of the Gramps Gtk implementation, please feel to call them out as I make the next ones. (Because, I will!) I'm ready to accept brutal honesty in expressed opinions.

So long as you criticize my Ideas, not me, the person; I believe that to be within bounds.

[jralls]

@emyoulation Thanks, but apparently that sentiment isn't universal.

[Nick-Hall]

@Nick-Hall perhaps it would be time to consider having a code of conduct (like we have for Gramps Web API) also for the Gramps main repo.

We already have a code of conduct. You can read it in our Discourse FAQ. This applies to the forum, mailing lists, IRC and GitHub. I'm not sure that we need something different for the Gramps Web API.

[Nick-Hall]

So long as you criticize my Ideas, not me, the person; I believe that to be within bounds.

I agree. Our code of conduct includes the statement: "But remember to criticize ideas, not people."

John's contributions are helpful and constructive. I don't think that any insult was intended.

[DavidMStraub]

Thank you @jralls, I do believe it was meant in a constructive way, sorry if I was overly sensitive. 🤗

[emyoulation]

Very gracious! Thank you for reconsidering.

Considering the amount of time that you are putting into Gramps Web development, perhaps it you should schedule and announce some "down days"? You have to be constantly exhausted.

Some scheduled "unavailabilty" might encourage a Gramps Web user (or more) to volunteer to help with support on those days.

I think that my primary value in this project has been to relieve a little of the support burden... particularly for repetitive questions from each batch of new users.

[Nick-Hall]

I don't think use as a library has ever been a Gramps design goal.

As far back as the Ecosystem definition in GEPS 012, there have been discussions about reusing Gramps core components. We have also always been open to collaborating with other open source genealogy projects.

I've considered making the Gramps core components accessible a long-term goal. This is probably best done by providing a library of core components rather than turning the Gramps application into a library.

The core library doesn't need to contain a database manager and family trees should be opened by path. The constants defined in const.py are application constants and not relevant to the library.

In addition to the core library, we could also consider providing a command-line only version of Gramps.

[jralls]

It certainly hasn't been a design goal originally, but it's a fact that Gramps is being used as a library today, and not just by some niche private projects. You (may or) may not like Gramps Web, but it's an official community project living in the gramps-project namespace and has > 600 stars on Github - more than Webtrees or any other genealogy repository (except Gramps).

Yes, I know about Gramps Web. It would be hard not to be since I see all of the Github gramps-project traffic. Gramps even documents an API for use outside of the app though the latest one indexed on the Developer wiki page is for 5.1 so it needs some updating. Great, but if it was properly separated from the application we wouldn't be having this discussion. Creating that separation is the way forward. Hacking on the app-specific stuff in the gen module to suit particular non-app projects only makes things worse.

[jralls]

The core library doesn't need to contain a database manager and family trees should be opened by path.

Would that work? The other end of the path is either a Berkeley or SQLite database.

This is probably best done by providing a library of core components rather than turning the Gramps application into a library.

Of course? ISTM the decision is whether the library should be in a directory of the gramps project or a separate project (gramps-lib ?) that gramps depends upon. The latter would be friendlier to the package managers.

[Nick-Hall]

Would that work? The other end of the path is either a Berkeley or SQLite database.

I think we only need the database plugins and the utility code in gramps.db.utils to load a database. This would need checking.

ISTM the decision is whether the library should be in a directory of the gramps project or a separate project (gramps-lib ?) that gramps depends upon.

Yes. That is the big question.

[bartfeenstra]

@jralls @Nick-Hall You both make some very good points that on their own, I cannot argue with. This is a hack. It's a hack because we've just been presented with a wonderfully shiny new major Gramps release that's going to stay with us for some time, but also intermediate deliverables are a thing. Technical debt can be worth it and managed well.

@Nick-Hall, you say "components". From my perspective it's essentially an "app" (stand-alone) versus a "library"/"component" that does not work on its own but can be imported by something else. And this is not an either/or thing. Even within a single application, most parts do not run stand-alone: they are libraries/components used by the wider app. In the majority of tests you want to load your own code as a component, not as a black-box application. Et cetera.

The question boils down to whether or not we want to allow other projects to import Gramps modules sooner (6.1) rather than later (7.0).

Considering Gramps' historical release frequency, not doing anything at all towards this until 7.0 will ensure that other projects won't be able to leverage some very, very useful parts of Gramps for the next several years.

On the other hand, lowering the barrier gradually, early, will allow the Gramps user base to grow, and will encourage more feedback and contributions.

[Nick-Hall]

@Nick-Hall, you say "components". From my perspective it's essentially an "app" (stand-alone) versus a "library"/"component" that does not work on its own but can be imported by something else.

One approach would be to make gramps-core a library component and gramps-cli and gramps-gui apps.

[bartfeenstra]

@Nick-Hall, you say "components". From my perspective it's essentially an "app" (stand-alone) versus a "library"/"component" that does not work on its own but can be imported by something else.

One approach would be to make gramps-core a library component and gramps-cli and gramps-gui apps.

It would. And I'd be absolutely happy with that for my particular use case. But what timeline and Gramps version are we talking about then?