Skip to content

Ecosystem Registry Architecture

Jump to bottom
Jay DeLuca edited this page Feb 2, 2026 · 1 revision

Terminology

Target Source Metadata

The raw metadata that exists at the source where telemetry is collected, typically alongside the project’s source code.

Examples:

  • Collector components: Each component directory contains a metadata.yaml file.
  • Java agent instrumentation: A single aggregated instrumentation-list.yaml file.

Registry Metadata

Metadata stored in a format and schema defined by this project, located in the ecosystem-registry directory.

Registry metadata is derived from target source metadata and may be transformed to support normalization, versioning, and long-term compatibility.

Ecosystem Registry Population

The ecosystem-registry is populated by automation within the ecosystem-automation directory.
This automation consists of components known as watchers.

A watcher is responsible for:

  • Monitoring a target project for metadata changes
  • Scraping and ingesting target source metadata
  • Transforming that metadata into the registry schema
  • Creating pull requests to update the registry when changes are detected

Watchers typically run as GitHub Actions on a scheduled basis (e.g., nightly) or in response to specific triggers.

Watcher Responsibilities and Requirements

When creating a new metadata watcher, the following requirements must be met:

Release and Version Awareness

  • The watcher must detect new releases of the target project and generate versioned inventory.
  • The watcher must maintain a snapshot version representing the current main (or equivalent) branch.
    • Snapshot data enables day-to-day change monitoring.
    • Snapshot data provides visibility into upcoming release changes.

Schema Evolution Support

  • The watcher must support evolving schemas at both:
    • The target source level
    • The registry level
  • If the target source schema is not versioned, the watcher must introduce an internal (“virtual”) schema version.
  • The watcher must, at a minimum, be able to map target schemas into the latest registry schema.

Schema Change Detection

  • The watcher must include automation to detect schema changes.
  • When a schema evolution is detected, the watcher should automatically open an issue to surface the change.

Backfill and Regeneration

  • The watcher must support backfilling and regenerating registry data for any historical version of the target source (assuming metadata exists).
  • This capability enables full registry rebuilds and simplifies recovery or migration scenarios.

Registry Output

= Items in the registry must be output in a deterministic ordering to avoid unnecessary updates

Watcher Structure and Ownership

Each watcher must be implemented as a separate module within the ecosystem-automation directory.

This structure:

  • Enables flexible code ownership
  • Allows for different technology choices per watcher
  • Scales as the registry expands to cover additional projects and ecosystems

Clone this wiki locally