[FEATURE] Support global resources support in multi-tenancy scenario

[zane-neo]

Persona

OpenSearch(OS) cluster manager: The person who owns the OpenSearch cluster and provide OS capabilities to normal users.
Normal User(NU): The person who uses OpenSearch capabilities provided by OS cluster manager.

What

Global resources are resources that can be accessed by all tenants, OS cluster manager can configure these global resources in their own data storage(Remote cluster/DynamoDB etc) with their own pipeline(CDK/management portal etc). These global resources are usually system level metadata, e.g. connector, model, agents in ml-commons or templates in flow-framework, and they're tenancy agnostic.

Why

1. With the global resources support we don't need to duplicate them in different tenants, saving storage resources for tenants.
2. Global resources can be managed by OS cluster manger so they can provide consistent features across all tenants.
3. Normal User doesn't have to be aware of global resources when they using the features provided by OS cluster.

How

Before an OpenSearch cluster spins up, OS cluster manager should configure all global resources first, all global resources will be attached with a global tenant id in data store. This global tenant id should also be configured into OS cluster settings when they starting the OpenSearch cluster.

There are two different cases should be handled in caching global resources:

1. Existing resources get updated.
2. New global resources are being added.

During the OS cluster nodes starting up, all the global resources will be fetched automatically and cached in cluster nodes memory, upon response(either success or failure), a scheduled task should be started to refresh the global resources cache with fixed interval, the task is to scan all the tables and fetch all global resources to cache. This approach can handle both cases above with a little bit latency, the interval should be a cluster setting configuration and customers can specify their own value.

When cache is not empty, read from cache first, if found return directly from cache. When cache is empty, read with actual tenant id from data store first, if not found, fallback to read with global tenant id from data store.

What solution would you like?

A clear and concise description of what you want to happen.

What alternatives have you considered?

A clear and concise description of any alternative solutions or features you've considered.

Do you have any additional context?

Add any other context or screenshots about the feature request here.

[dbwiddis]

If any exception caught during global resources fetching, the error will be thrown and stopping the cluster startup.

This could entirely kill a cluster during an upgrade/upscale/etc., or a node after bouncing the OS process/rebooting, all based on an external factor, with no ability for the cluster to self-heal.

How robust is the storage of global resources?

[zane-neo]

For remote data source case like DynamoDB or remote cluster, this should be fine as the error either comes from network issue or incorrect global resource configuration(e.g. unable to parse to json), the early stop can remind OS cluster manager to investigate on this as soon as possible.

But for local cluster, you're right, the indices might not ready yet and throwing exception during node startup could cause cluster failed to start and it's not recoverable. I'll change the issue details to cover this case.

[dbwiddis]

There are two different cases should be handled in caching global resources:

1. Existing resources get updated.
2. New global resources are being added.

What is the expected frequency of these updates? I can see updating models on a ~monthly or longer basis, or more frequent agent/prompt updates?

The existing code is very heavy on querying all the DDB tables to identify resource changes. How can we reduce this load? Having a single table containing a "last updated time" (perhaps on a per-table basis) might reduce this load: new startups query everything, but sheduled updates only need to make a few queries.

the interval should be a cluster setting configuration and customers can specify their own value.

You say customers don't need to be aware of global resources, so why would they do this? Also with the current implementation I don't see how individual tenants get a different refresh value than the entire cluster.

[zane-neo]

The existing code is very heavy on querying all the DDB tables to identify resource changes. How can we reduce this load? Having a single table containing a "last updated time" (perhaps on a per-table basis) might reduce this load: new startups query everything, but sheduled updates only need to make a few queries.

I prefer to cache on demand as the discussion in the PR.

You say customers don't need to be aware of global resources, so why would they do this? Also with the current implementation I don't see how individual tenants get a different refresh value than the entire cluster.

What I said is normal user doesn't need to be aware of global resources, cluster owner still aware of this.