Data sync prerequisites checks to build intermodule relationships

[heryxpc]

Some intel modules and ontologies require / have interdependencies with other intel modules to get a full graph of relationships.
A few examples are mapping the legacy :CVE nodes to Github repositories via :SemgrepSCAFinding or to a running container in AWS via TrivyImageFinding.

Which would be similar if using the CVEMetadata ontology focused module. To do the enrichment, properly would require syncing AWS, then Trivy and finally CVE Metadata.

Other example would be matching Human to Okta users to AWS assumable roles via (:Human)-[:IDENTITY_OKTA]->(:OktaUser:UserAccount)-[:CAN_ASSUME_IDENTITY]->(:AWSSSOUser)-[:ASSUMED_ROLE_WITH_SAML]->(:AWSRole), requiring first Okta and then AWS.iam syncs.

This adds a required sequence:

1. Sync module A
2. Sync module B creating relationships with A
3. Sync module C creating relationships with B.

Running all syncs in a single shot can be very memory and time consuming if the data is very large (e.g. >500k nodes + > 1Mi relationships). This leads to a multi step process that drives the strategy of "eventually consistent" after sync's ran in the right sequence.

Currently, there is no mechanism to enforce the interdependencies, as Cartography has no orchestration layer per se.

However, a mechanism to check or at least do a "best effort" to ensure interdependencies are met to ensure the right connections for the right question exist could:

• Simplify the usability of new comers to answer complex questions in the graph (which users can access which data?, sync user data A then data storage B)
• Hint on the synchronization requirements for extended connected data (sync(A,B), sync(BC) to connect A -> B -> C).
• Add a data health check (if missing r in (:A)-[r:HAS]->(:B) then sync(B) might be missing relationship attributes).

Building a DAG/orchestration layer might be too complex, but having prerequisites checks for relationship building could flag if the relationship will be created or not before the sync starts.

One possible path to follow is using : ModuleSyncMetadata checking the existence and freshness (lastupdated) for the prereq sync. For example, to check if an Okta user can assume a role to write an S3 bucket, we could query:

WITH 'account_id' AS account
MATCH (n:SyncMetadata)
WHERE (n.grouptype = 'OktaOrganization' AND n.syncedtype = 'OktaOrganization')                    // Okta (org-scoped, not per AWS account)
   OR (n.grouptype = 'AWSAccount' AND n.groupid = account AND n.syncedtype IN ['S3Bucket','AWSPrincipal'])  // S3 + IAM for this account
RETURN
  CASE n.syncedtype
    WHEN 'OktaOrganization' THEN '1. Okta'
    WHEN 'S3Bucket'         THEN '2. S3'
    WHEN 'AWSPrincipal'     THEN '3. IAM'
  END                                        AS sync_step,
  n.grouptype                                AS module,
  n.groupid                                  AS scope,
  n.syncedtype                               AS synced_type,
  datetime({epochSeconds: n.lastupdated})    AS last_synced,
  datetime({epochMillis: n.firstseen})       AS first_seen
ORDER BY sync_step

We could get if data exists and how recent is from a prereq sync. Then, count the number or relationships created and add a comparison against a benchmark or threshold. This could be done further with stats or rule facts to give a predictive answer based on a scalar.

Happy to hear your thoughts before creating an issue and discuss implementation details.