|
| 1 | +# Streaming |
| 2 | + |
| 3 | + |
| 4 | +Streaming plugins use grpc streams to allow the plugin to send data immediately or after a certain period of time instead of on an interval governed by Snap. |
| 5 | +Streaming by snap enables - |
| 6 | +* Improved performance by enabling event based data flows |
| 7 | +* Runtime configuration controlling event throughput |
| 8 | +* Buffer configurations which dispatch events after a given duration and/or when a given event count has been reached |
| 9 | + |
| 10 | +Currently there are two plugins that support streaming - [snap relay](https://github.com/intelsdi-x/snap-relay) and [snap-plugin-collector-rand-streaming](https://github.com/intelsdi-x/snap-plugin-lib-go/tree/master/examples/snap-plugin-collector-rand-streaming). |
| 11 | + |
| 12 | +# Configuration options |
| 13 | +MaxCollectDuration and MaxMetricsBuffer are two configuration options that can be set through streaming task manifest or flags. |
| 14 | +* MaxCollectDuration sets the maximum duration between collections before metrics are sent. It is always greater than 0 and defaults to 10s which means that after 10 seconds if no new metrics are received, the plugin should send whatever data it has in the buffer. |
| 15 | +* MaxMetricsBuffer is the maximum number of metrics the plugin is buffering before sending metrics. It defaults to 0 which means the metrics are sent immediately. |
| 16 | + |
| 17 | +# Streaming task schedule |
| 18 | +``` |
| 19 | +--- |
| 20 | + version: 1 |
| 21 | + schedule: |
| 22 | + type: "streaming" |
| 23 | + workflow: |
| 24 | + collect: |
| 25 | + metrics: |
| 26 | + /random/integer: {} |
| 27 | + config: |
| 28 | + /random/integer: |
| 29 | + MaxCollectDuration: "6s" |
| 30 | + MaxMetricsBuffer: 600 |
| 31 | +``` |
| 32 | + |
| 33 | +# Streaming configuration flags |
| 34 | +Below is an example of the how to run the snap-relay using the configurable flags. |
| 35 | +1. Start the Snap daemon: |
| 36 | +* Run |
| 37 | +``` |
| 38 | +$ snapteld -l 1 -t 0 |
| 39 | +``` |
| 40 | +The option "-l 1" is for setting the debugging log level and "-t 0" is for disabling plugin signing. |
| 41 | + |
| 42 | +2. Start the relay using flags: |
| 43 | +* Run snap-relay plugin |
| 44 | +``` |
| 45 | +$ go run main.go --stand-alone --stand-alone-port 8182 --log-level 5 --max-collect-duration 10s --max-metrics-buffer 50 |
| 46 | +
|
| 47 | +Output: Preamble URL: [::]:8182 |
| 48 | +``` |
| 49 | + |
| 50 | +3. Obtain meta information: |
| 51 | +* Curl preamble |
| 52 | +``` |
| 53 | +$ curl localhost:8182 |
| 54 | +{"Meta":{"Type":3,"Name":"plugin-relay","Version":1,"RPCType":3,"RPCVersion":1,"ConcurrencyCount":5,"Exclusive":false,"Unsecure":true,"CacheTTL":0,"RoutingStrategy":0,"CertPath":"","KeyPath":"","TLSEnabled":false,"RootCertPaths":""},"ListenAddress":"127.0.0.1:54541","PprofAddress":"0","Type":3,"State":0,"ErrorMessage":""} |
| 55 | +``` |
| 56 | + |
| 57 | +4. Start client in snap-relay: |
| 58 | +* Run client (from a different terminal) using the listen address obtained from the previous step |
| 59 | +``` |
| 60 | +$ go run client/main.go 127.0.0.1:54541 |
| 61 | +``` |
| 62 | + |
| 63 | +5. Load plugin: |
| 64 | +``` |
| 65 | +$ snaptel plugin load http://localhost:8182 |
| 66 | +Plugin loaded |
| 67 | +Name: plugin-relay |
| 68 | +Version: 1 |
| 69 | +Type: streaming-collector |
| 70 | +Signed: false |
| 71 | +Loaded Time: Tue, 05 Sep 2017 17:41:42 PDT |
| 72 | +``` |
| 73 | + |
| 74 | +* List the metric catalog by running: |
| 75 | +``` |
| 76 | +$ snaptel metric list |
| 77 | +NAMESPACE VERSIONS |
| 78 | +/intel/relay/collectd 1 |
| 79 | +/intel/relay/statsd 1 |
| 80 | +``` |
| 81 | + |
| 82 | +6. Create a task manifest for snap-relay (see [exemplary files](https://github.com/intelsdi-x/snap-relay/tree/master/examples/tasks)): |
| 83 | +``` |
| 84 | +--- |
| 85 | + version: 1 |
| 86 | + schedule: |
| 87 | + type: "streaming" |
| 88 | + workflow: |
| 89 | + collect: |
| 90 | + metrics: |
| 91 | + /intel/relay/collectd: {} |
| 92 | +``` |
| 93 | + |
| 94 | +7. Create a task: |
| 95 | +``` |
| 96 | +$ snaptel task create -t collectd.yaml |
| 97 | +Using task manifest to create task |
| 98 | +Task created |
| 99 | +ID: c168b992-8aaf-4eec-8e3c-883510962789 |
| 100 | +Name: Task-c168b992-8aaf-4eec-8e3c-883510962789 |
| 101 | +State: Running |
| 102 | +``` |
| 103 | + |
| 104 | +# Metrics exposed by streaming collectors |
| 105 | +Below are some of the metrics collected by the streaming plugins currently: |
| 106 | +``` |
| 107 | +/intel/relay/collectd |
| 108 | +/intel/relay/statsd |
| 109 | +/random/integer |
| 110 | +/random/float |
| 111 | +/random/string |
| 112 | +``` |
| 113 | + |
| 114 | + |
| 115 | + |
| 116 | + |
0 commit comments