Suggestions: Context Translation Middleware

[pvoznyuk]

Enhancement Request

Use Case:

Suppose there is an existing ecosystem of applications that communicate with each other via DesktopAgent. The context for the fdc3.instrument type includes the id.ISIN property. One day, a third-party application is to be integrated into the ecosystem and communicate with the existing applications via DesktopAgent.

The problem is that this application doesn’t support id.ISIN but instead supports id.FIGI.

Refactoring all existing applications to add id.FIGI to every context broadcast is expensive, so it would be helpful if the contexts could be converted automatically according to defined rules.

Workflow Description

Context Translation Middleware: Middleware within DesktopAgent that automatically converts contexts based on defined rules (e.g., context type) during broadcasting.

• This conversion can be synchronous (in-place) or asynchronous (e.g., by requesting a service to transform the context).
• There can be multiple middleware.

Error Handling: If there are any cases where the translation is not feasible, DesktopAgent should handle this so developers can log the situation and alert the administrators. This ensures visibility for unsupported context translations and reduces potential issues during runtime.

Workflow Examples

const contextTransform = async (context) => {
    if (context.type === 'fdc3.instrument') {
        if (context.id.ISIN && !context.id.FIGI) {
           const FIGI = await convertISINtoFIGI(context.id.ISIN) // a fetch request to some service
           if (!FIGI) {
              throw new Error(`Cannot convert ISIN ${context.id.ISIN} to FIGI for fdc3.instrument`)
           }
           return { 
               ...context,
               id: {
                  ...context.id,
                  FIGI,
               }
           }
        }
    }
}

getAgent({
   middleware: [
      contextTransform
   ]
})

[novavi]

This is an interesting proposal. Someone in my firm raised this same issue to me last year e.g. whether there are standard mechanisms in FDC3 to handle this type of requirement. I've also had similar discussions more recently relating to translation not so much between two different standard identifiers, but between standard and internal/proprietary identifiers. Just to open up the just discussion a bit, here's a few thoughts I had when reading the above:

• Outbound contexts. You mentioned broadcasting, but it's worth being more specific. Was the intention that this should apply not just to fdc3.broadcast(), but to everything responsible for outbound context e.g. fdc3.raiseIntent(), and fdc3.raiseIntentForContext() as well?
• Broadcasting / Listening. You mentioned specifically broadcasting, but not listening. One could ague there isn't necessarily a 'right' place to do this - it really depends on which app(s) are within your control. I have done something similar before on the listener side (in target app) rather than on the broadcast side (in source app) this was simply because only the target app was under my control. (Of course this relied on using a context resolver inside multiple listener functions though, which was obviously suboptimal.)
• IntentResult. In addition to considering contexts that are dealt with by fdc3.broadcast(), fdc3.raiseIntent(), fdc3.raiseIntentForContext(), fdc3.addContextListener() and fdc3.addIntentListener(), there's also the issue of IntentResult i.e. results that an intent handler can return to communicate back to the app that raised the intent (which can also be contexts, and therefore potentially also in scope for transform).
• Multiple transforms. If you open up the possibility of doing this on the listener side as well as the broadcast side, and if you don't control all of the apps (e.g. some of them provided by other vendors) you might have some situations beyond your control where the middleware was configured for both the source app(s) and the target app(s). If there were situations where a context transform might run more than once, it would probably make sense to think more in terms of enrichment rather than transform.
• Guardrails. It's probably worth thinking about what what should be permitted and what should be restricted by any middleware of this nature:
  • It seems perfectly valid to enrich a context by looking up and adding in additional identifiers, as the original intention of the source app and the existing properties would remain unchanged. But if you have a situation where the context type property could potentially be updated by a function provided to the middleware (which is technically possible in the example above) then this is a whole different thing.
  • If an App Directory record indicated that an app broadcasts or raises or listensFor an fdc3.something context but the middleware caused it to actually send/receive an fdc3.somethingElse context instead, this would change many assumptions apps routinely make about other apps in their platform/ecosystem.
  • I think caution is required for anything that opens up the possibility to change the context type property on the fly (as opposed to simply enriching context) as this would be a far bigger change with significant implications to the way FDC3 works e.g. could you still use fdc3.findIntent() or fdc3.findIntentsByContext() to reliably look up apps and app instances that are registered to handle a particular context, if you have scenarios where context types could change between source app and target app? There are related implications for Intent Resolvers e.g. when a target app is not specified.
  • There are practical limits to what guardrails could be applied if the middleware could accept arbitrary transform functions. You could potentially have the code that handles the middleware verify that your transform function has not removed or tampered with properties from the original context after the fact and then raise an error if this happened - but that would be a very clunky approach. Another possibility would be to change the concept of the transform function to instead emit an object containing just the additional properties / sub-properties / sub-sub-properties / etc. and then have the code that handles the middleware perform a deep merge of the original context and the additional context. And the deep merge would have to be implemented in such a way that it avoided updating existing property values, and only added new properties. Assuming guardrails are required, this area needs a lot more thought / discussed to arrive a good solution which supports flexibility but also provide sensible guardrails. [1]
• Provision to configure/transform in DA proxy (running in app) versus inside DA itself. Though this pattern does require all apps to be changed to update their getAgent() invocation (with the contextTransform quite likely coming from a shared npm package consumed by multiple apps) I think I can see why you're suggesting that transform should be controlled in the DA proxy at app-level. Prior to FDC3 v2.2, such transform would more likely have entailed configuring centrally at DA-level i.e. inside the vendor DA, rather than in a proxy running at app-level. Which of course would be a vendor DA implementation issue (and not covered by the FDC3 specification because it does not mandate things around DA configuration / bootstrapping). But with the advent of FDC3 2.2, configuring and performing the transform at DA-level would have negative implications for how agnostic your app could be to different DA vendors and different DA configurations. For example, at runtime any app using the getAgent() method might use a DA with expected middleware, but could end up using a DA with unexpected middleware or no middleware. (However, it's worth noting that extending the DA proxy to support for middleware does not prevent any DA vendor providing transform functionality that runs inside their hub/broker. If they wished, an app vendor could configure the DA proxies used by their apps to use no middleware at all, and then utilize a DA vendor's transform functionality if available - and if the app vendor did not care about making their apps DA-agnostic.)
• Sub-agents. Presumably sub-agents (expected to be part of FDC3 v2.3) would implicitly inherit the middleware provided at the original getAgent() level? Should it also be possible to provide sub-agent-specific middleware to a sub-agent at creation time, or does it not make sense to allow this?
• Error handling. Assuming your apps were dependent on the contextTransform succeeding for the cross-app workflow to operate properly (if they were not dependent on this why would they even use it?) then this sounds like it needs an addition to multiple different error enums (exactly which enums requires being more specific about the scope of where the middleware would apply e.g. some of the questions raised above). Actually it would likely be multiple additions to the enums, because there could be multiple error conditions e.g. the contextTransform could throw an exception, or it could just timeout, so it would probably make sense to have something like ContextTransformFailed and ContextTransformTimeout at a minimum.
• Tracing. If transforms were performed on outbound contexts by a source app's DA proxy, the transformed context (as opposed to the original context) would be shown in any traces captured by a vendor DA. But if transforms were performed on inbound contexts (which to be fair I know was not explicitly in the scope of the proposal above) presumably any vendor DA traces would only capture the original context. This would likely have interesting implications for debugging, so it would be worth working through all possible scenarios and consider what could be done to ensure tracing was still usable. Keeping the scope of the proposal limited to outbound contexts only would certainly make this aspect easier, but as I suggested above I think that might also limit the effectiveness of the proposal because as an app vendor you can only configure a transform in the apps you control - and those apps could be target apps rather than source apps.

[1] Just to be more specific about one of the suggestions above on guardrails and changing the concept of the transform function to instead emit just the additional properties, what I was getting at was something along the lines of:

Original context (pre-middleware execution):

{ 
    type: "fdc3.instrument",
    id: {
        cusip: "594918104"
    }
}

Additional properties (return value from enrichment function provided to middleware executor):

{ 
    id: {
        isin: "US5949181045"
    }
}

Enriched Context (post-middleware execution):

{ 
    type: "fdc3.instrument",
    id: {
        cusip: "594918104",
        isin: "US5949181045"
    }
}

But given that you could not prevent the enrichment function from returning an object also containing existing properties, the deep merge would have to ensure it only added new properties and did not update existing property values. On this basis, the enrichment function would effectively be free to return an extended context (containing both existing and new properties) which would likely be easier to author. But the key point is that the middleware handler would not simply use the value returned from the enrichment function as-is, it would apply this very specific (guarded) deep merge.

Of course, there are downsides to the above approach - hence I think that if there was an intention to make the proposal support guardrails, a number of different approaches and patterns would need to be put on the table, evaluated, and discussed.