Deep dive: Import Export

Import/Export

The import/export mechanism hinges around 3 key components:

1. An ImportController that manages Import CRD instances
2. An ExportController that manages Export CRD instances
3. A REST-based SubscriptionBoard service that exposes dynamic routes to PEs

The user scenario below explores, step by step, the role of these components in resolving import/export connections.

User scenario

For this deep dive, we consider the following user scenario:

• User submits Job A with a single exported stream, named streams0
• User submits Job B that imports the named stream streams0

User submits Job A

Step 1 : user submits the new Job

$ kubectl apply -f JobA.yaml

Step 2 : Kubernetes notifies the Job controller

Upon addition of the Job A resource, the job controller inspects the AADLs generated by the pipeline for imports and exports. In the case of Job A, the import set is empty.

Then, the job controller processes the import and export sets. For each export the controller calls ExportFactory::addExport to create an Export CRD instance. In the case of Job A, a single Export instance is created.

Step 3 : Kubernetes notifies the Export controller

Upon addition of the new Export resource, the export controller calls ImportExportBroker::add. For this new Export, there is no Import to be matched so nothing is done.

User submits Job B

Step 1 : user submits the new Job

$ kubectl apply -f JobB.yaml

Step 2 : Kubernetes notifies the Job controller

Upon addition of the Job B resource, the job controller inspects the AADLs generated by the pipeline for imports and exports. In the case of Job B, the export set is empty.

Then, the job controller processes the import and export sets. For each import the controller calls ImportFactory::addImport to create an Import CRD instance. In the case of Job B, a single Import instance is created.

Step 3 : Kubernetes notifies the Import controller

Upon addition of the new Import resource, the import controller calls ImportExportBroker::add. The method will match the new Import with all existing Export. In this scenario, the Export created by Job A is matched.

Step 4 : the broker updates the subscription board and notifies the PEs

Since there is a match, the ImportExportBroker updates the subscription board with the new route information for the importing PE. Then, it sends a datagram notification to the target PE.

In the meantime : PEs poll the subscription board

PEs use a pull model to update their dynamic routes. This model is implemented using a periodic polling mechanism that fetches routes for a PE every minute.

Route updates only impact PEs with Import operators. Job A has no Import operator, so its dynamic routes are not updated.

For Job B, once the subscription board is updated, the notification sent to the PE is received. The content of the subscription board for that PE is then fetched and compared to its local copy.

Finally, the modifications are translated into route ADD, DEL and UPDATE signals to the PE.

Reference

Bootstrapping

The bootstrapping section for imports and export is located in the job controller.

Import

1. Collect input ports that contain imported streams
2. Create an Import CRD for each of the imported stream

Export

1. Collect output ports that contain exported streams
2. Create an Export CRD for each of the exported stream group

Custom resource definitions

Import

The Import CRD contains the following attributes:

1. Its job ID
2. Its PE ID
3. Its port ID
4. Its port index
5. Its ImportedStreams object

Export

The Export CRD contains the following attributes:

1. Its PE ID
2. Its port ID
3. Its ExportedStreams object

Matching

Import/Export matching is handled by a synchronized Broker object.

When the Import controller is notified of the creation of an Import CRD instance, Export are matched through the broker. When the Export controller is notified of the creation of an Export CRD instance, Import are also matched through the broker.

Matched Import and Export and exposed to PEs using a REST service. The service is updated by the broken each time an Import or an Export is matched (here and here).

Route update

PEs use a pull model to update their dynamic routes. This model is implemented using a periodic polling mechanism that fetches routes for a PE every minute.

In addition to that, PEs are notified of a match using a datagram notification. This notification is unreliable by design.

Programming interface

PEs use the REST interface to implement filter and subscription changes from an operator (eg. adding properties).

Reliability

If the instance controller restarts after a failure, the matching is automatically recomputed using the values from existing Import and Export instances.