Knowledge Graph Storage Component

[cbizon]

On our Miro board, we have a component labeled "KG-Hub". This is the location where KGX or similar files created by data ingestion are held and transferred to the data access working group.

We need to make a few decisions around this component.

First, what is it called? KG-Hub is a specific tool which we might or might not use (see below). So I propose that we call this component something else for the time being until we decide what it is to avoid confusion. Maybe something simple like KGStore.

Second, how is KGStore implemented. I think we're expecting that these files will be kept in some form of cloud storage such as an S3 bucket (or whatever is appropriate for the cloud provider that ITRB is using).

But on top of that, do we want to use anything around that bucket? I am aware of 3 options, there are probably more:

1. Nope, just a regular old bare bucket. We make some rules about the structure of the filesystem. Very low overhead/upkeep but also provides the least support
2. KG-Hub https://github.com/knowledge-Graph-Hub. This tool comes from LBNL so we have institutional knowledge. Handles metadata and references graphs that are stored in other places (e.g. our bucket).
3. lakefs (https://lakefs.io/) gives you git-like versioning and actions on the data. Can be backed with numerous storage types. We have some experience with this from a different project (ProtoOKN).

For the group, I have a few questions:

@sierra-moxon (?) can you say a couple of words about what features kghub could provide?
@YaphetKG could you add a comment about what lakefs could provide?

others - are there other options in this space that we should consider?

[YaphetKG]

Lakefs as @cbizon mentioned, is a git like implementation designed for data files. It provides versioning layer over a backend storage such as s3, local disk , azure blob etc.

Some of the data management features include:

• Branches
• Branch rules : allowing protection on certain branches to make sure data consistency
• Tags : Lets users tag commits as snapshots

Features related to workflows:

• Provides hooks on certain events , events such as pre-commit, post-commit , on merge , on tag creation etc...
• These hooks can be webhooks or lua scripts. Webhooks can make api calls to other systems to react to these events.

In our ProtoOKN experience one way we are using lakefs is to automate deployment. We have several teams that upload their data to a lakefs repository they one, and merge to a standard main branch . That event triggers a webhook that handles conversion / validation / graph characterization . Once that is done the system notifies owners that that all is complete so they can take a snapshot of this (basically tag ) , this event triggers another flow that automatically updates the servers that serve these data with this new version .

This is all great but one thing that might be good to consider also is how users would interact with the system. Lakefs has an authentication layer which requires users to login , to be able to upload data etc... which can be a good thing depending on how open we want graph submission provisioning credentials might be something to consider.

[cbizon]

Thanks @YaphetKG for this description. On the authentication front, does this imply that uploads must be done by hand through the UI?

[sierra-moxon]

I think Lakefs looks nice!

KG-Hub technically does include a single storage location to host KGX files, but we would rather folks host their own KGX files and point to them from the KG-Registry.

kg-registry is an important piece to this puzzle in that it provides a nice suite of metadata and functionality for people wanting to find a KG or suite of KGX files: KG-Registry - the deployed site is here and has a good search functionality: https://kghub.org/kg-registry/. KG-registry also has ties into the information-resource-registry (https://github.com/biolink/information-resource-registry) and would be a decentralized, low-cost way for folks to find and reuse KGX files in a modular ingest architecture.

In terms of lakefs, I have also been encouraged to look at delta io on the LBL side as several larger projects here use that platform/tooling (my reading of this is that is more than what our original use cases here intend - ours are a simple storage framework for sets of files, so delta.io might not be the fit we're looking for for this ticket).

I have also been looking at RO-crate: https://www.researchobject.org/ro-crate/, another popular framework in our sphere for bundling files and metadata. This sits slightly outside this particular issue's questions, too, in that it handles the bundling of files and metadata in a standard format, not necessarily dictating the physical storage of the crate. However, I would like to think about/discuss what "metadata" lakefs give us by default and how that overlaps or plays nicely with the metadata we want to include for each KGX bundle. For example, we might have the version of the lakefs data release, the version of the data ingested, the version of the KGX release of the data ingested, the version of Translator this bundle is used for, etc. We need all this metadata (and more!). Should it all be in one file associated with the ingest/source or spread between a few locations?

Final thing: we did stand up the Knowledge Graph Exchange which was a hosted platform for storing KGX bundles in the last phase of translator (KGE) - @RichardBruskiewich - could you please comment on the benefits of central storage (and also the challenges you encountered in the last phase)?

@justaddcoffee @caufieldjh

[sierra-moxon]

Oh I also have been encouraged to use zenodo for storage, but with a bit of trepidation as it becomes a bit unstable as file sizes get larger.

[RichardBruskiewich]

I think we're expecting that these files will be kept in some form of cloud storage such as an S3 bucket (or whatever is appropriate for the cloud provider that ITRB is using).

(KGE commentary, Part 1) The original Knowledge_Graph_Exchange_Registry code base was AWS S3 based, albeit a code base which was very rough around the edges... many practical issues were resolved but some not so much so. A tiny team were involved in the implementation, which was self limiting, but perhaps if other developers were to contribute their expertise, some of the code could be adapted or used as a modest source of ideas on a Phase 3 implementation of KGStore (e.g. some of the discussion about metadata?)

That said, both KG-Hub and Lakefs seem to be much better designed, more robust and implemented by larger development teams with broader institutional support, so....

[RichardBruskiewich]

Final thing: we did stand up the Knowledge Graph Exchange which was a hosted platform for storing KGX bundles in the last phase of translator (KGE) - @RichardBruskiewich - could you please comment on the benefits of central storage (and also the challenges you encountered in the last phase)?

(KGE commentary, Part 2)
The Knowledge Graph Exchange was an AWS EC2 / S3 / etc. based application written in Python, with an AIOHTTP-based web front end, calling an OpenAPI web service API which accessed AWS S3 storage.

The original design was systematically derived from the KG Exchange working group deliberations, captured on the main Knowledge Graph Exchange Roadmap web site.

Certainly, the potential core benefit exists of one system being engineered to solve a common community data sharing need and centrally indexing available data for access. On the one hand, the system becomes a single point of failure (this feels like a manageable problem though). On the other hand, data access won't vanish is distributed archives of such data go offline or access performance is hampered for technical reasons or for want continued funding (or policy changes). Properly designed, a central archive can optimize FAIR data access for a project whose fundamental mantra is community-wide sharing of data. The KGE Registry was designed with that mandate in mind.

However, when implementation was started, the development team was rather small, which was itself a primary challenge limiting delivery of the system. Although KGX file sharing on AWS sounds simple on the surface, implementation details suggest otherwise. If such a resource has high value within Translator (i.e. see diminished relevance below, during Phase 2) and needs to be commissioned in a timely fashion, then having an adequately sized team established up front implement it, or drawing upon an existing project (i.e. KG-Hub, Lakefs, delta.io, etc.), is a necessity, not a luxury. That said, the project received excellent technical feedback from Stephen Ramsey (RTX-KG2) to improve KGX data upload performance, and from Jeff HHK (of the MediKanren team), the latter highlighting technical concerns about direct API KGX file access for their team, although follow-up was ultimately limited on that front.

Digging a bit deeper into the Knowledge_Graph_Exchange_Registry project itself, there are perhaps a few observations worthy of note.

• It is often hard to anticipate ahead of time when a given choice of core technology for a project will put up unexpected barriers. The choice of AIOHTTP resulted in at least one tricky technical issue significantly into implementation. The details of the curious issue are somewhat forgotten now but perhaps it was allegedly to be resolved in a subsequent release of the software, patiently awaited prior to the stopping of work on the KGE Registry.

• The huge size of some Translator KGX datasets required greater flexibility in EC2 / EBS provisioning for data loading into AWS S3 buckets, and KGX validation, an issue which was progressively being resolved before work on KGE Registry was abandoned.

In many respects, the KGE application was evolving steadily away from "half baked cake" status to greater robustness and completeness in implementation before work upon it was stopped. I stand to be corrected on this point, however, I don't recall any "show stopping" issues which would have prevented its eventual relative maturity in implementation and delivery: it appeared to be in a relatively usable "beta" state of release when the work was stopped.

However, the underlying inter-team motivation for KGX file sharing appeared to diminish dramatically during Translator Phase 2 over time, which was the primary reason continued work on the resource was eventually abandoned in favour of higher priority Translator SRI work. The arrival of the LBNL KG-Hub on the scene also added impetus to the decision to stop the KGE work, although further discussions in Phase 2 didn't yet arrive at the adoption of KG-Hub for KGX file sharing use cases within Translator.

[YaphetKG]

Thanks @YaphetKG for this description. On the authentication front, does this imply that uploads must be done by hand through the UI?

The uploads can be done in several ways, including lakefsctl (a cli utility for lakefs) , python , s3 compatible tool such as aws s3 cli tool , or rclone...

The but for all of the tools i tried to list, they would need to be supplied with the auth credentials

[gaurav]

The uploads can be done in several ways, including lakefsctl (a cli utility for lakefs) , python , s3 compatible tool such as aws s3 cli tool , or rclone...

Rclone has good support for sync operations on LakeFS, such as if you have a bunch of files in a local directory and you want a directory in a LakeFS repository to reflect those files exactly without overwriting or re-uploading any files that haven't changed. I did need to commit these changes separately, but it was easy to do that with curl. We used environmental variables and Kubernetes secrets to store LakeFS authentication information, but it looks like you can also store that information in an rclone.conf file.

[saramsey]

I saw in the agenda for the Engineering Committee meeting (on 4/23/2025) today that I should post in "Issue 99" regarding thoughts about file formats for handoff between the Data Ingest WG and the Retriever WG. Here are my thoughts (FWIW!).

For the file format(s) to be handed off between the Data Ingest WG and the Retriever group, might I suggest considering a simple three-file format (shown for the example of a DrugBank ingest):

drugbank-kgx-nodes.jsonl
drugbank-kgx-edges.jsonl
drugbank-kgx-metadata.yaml

I like the idea of a separate metadata file (YAML is fine, and I suppose preferred insofar as it allows comments) which can be scriptomatically read first, which would then give the version number and other relevant metadata. Provided we keep the metadata schema simple and document it.

Hope this comment isn't out of turn and/or inappropriate for this GitHub issue.

[saramsey]

Whatever we do, I'd just like to strongly advocate for continuing JSON-lines as a fully-supported file format for KGX interchange of knowledge graphs. It nicely allows for streaming ingests over HTTP. There are real use-cases where we don't want to download the JSON-lines file, we just want to read it via HTTP and process it in chunks.

[saramsey]

Regarding how to submit KGX graphs in the new system:

To me, "easy submission" means providing a well-documented, simple, programmatic method to submit KG files with their metadata into the archive (whatever we end up calling this central archive); as well as (ideally) an easy and intuitive way to do it by the submitter writing a short YAML file or other descriptive document and using github or a web browser to submit it.

[saramsey]

After researching this a bit more, I believe I was wrong in thinking that chunked processing of remote JSON-lines data files via HTTP requires Transfer-Encoding: chunked. I now think that it does not. I think the "Transfer-Encoding: chunked" is for applications like live-streaming audio/video where there is not a well-defined file size on the server side. In any event, the ability to be able to load a KG data file via HTTP and process it in chunks without having to download the whole file all at once, is SUPER SUPER helpful. I would go so far as to say essential. Just my $0.02. Apologies for adding to the confusion in the April 9th meeting.