Installing a Simple Data Pipe Connector

Introduction

The Simple Data Pipe example application includes connectors for a couple data sources. However, additional connectors (e.g., Reddit AMA, SugarCRM) are available to add into the Simple Data Pipe. If a particular connector cannot be found it is also possible to create custom connectors for specific data sources.

Once the desired connector is found or created, installing it into the Simple Dat Pipe is straightforward. This tutorial will step through how to install a connector into a deployment of Simple Data Pipe.

Depending on the Simple Data Pipe environment, there are two options for installation a connector:

• The remote edit approach where updates are made directly in the IBM Bluemix DevOps Services Web IDE then deployed.
• The local edit approach where updates are made locally then pushed and deployed to the IBM Bluemix environment.

Installing a Connector Remotely

To update a Simple Data Pipe deployed in IBM Bluemix remotely. The application would need to have been cloned and available in the Git repository that is part of IBM Bluemix DevOps Services. This is the case, if installation of the Simple Data Pipe was done via the Deploy to Bluemix button. To install a connector in this scenario:

1. Log into IBM Bluemix

2. Go into the Simple Data Pipe application Overview page

3. Go into the IBM Bluemix DevOps Services Web IDE

   • If present, click on EDIT CODE in the Overview page to launch the Web IDE
   • Else, log into IBM Bluemix DevOps Services and go into the Simple Data Pipe project

4. Once in IBM Bluemix DevOps Services, click on EDIT CODE to enable live edit

5. Click on package.json

6. Edit the package.json to add the connector to the dependencies list. For example:

   "pipes.connector.reddit": "https://github.com/ibm-cds-labs/pipes.connector.reddit.git"
   

   Note: Be sure to follow proper JSON syntax when editing package.json

7. Save the changes ( > File > Save with the Web IDE)

8. Click on the Deploy the App button (right arrow)

The application will be rebuild and deployed. Once deployed, Simple Data Pipe shoud now include the new connector.

Installing a Connector From Local

A Simple Data Pipe deployed in IBM Bluemix which does not include a Git repository that is part of IBM Bluemix DevOps Services requires a local clone of the Simple Data Pipe repository. The Cloud Foundry command line interface is also required. To install a connector in this scenario:

From a command line

1. Switch into Simple Data Pipe repository directory: cd pipes

2. Install the connector with npm. For example:

   npm install https://github.com/ibm-cds-labs/pipes.connector.reddit.git
   

3. Log into Bluemix with the appropriate [org] and [space]:

   cf login -a https://api.ng.bluemix.net -o [org] -s [space]
   

4. Push the Simple Data Pipe application:

   cf push -f manifest.yml
   

Once completed, the updated Simple Data Pipe should be available within IBM Bluemix with the new connector.

Conclusion

Additional Connectors can easily be added as needed to the Simple Data Pipe using either of the above approaches. For the connectors to be properly recognized and made usable within the Simple Data they will need to makes use of the pipes-sdk Connector API.

See the Simple Data Pipe GitHub repo, the pipes-sdk GitHub repo, and the Simple Data Pipe landing page for additional information and resources.