diff --git a/README.md b/README.md index 39761f4..4895d1d 100644 --- a/README.md +++ b/README.md @@ -51,6 +51,7 @@ This repository provides examples demonstrating how to use Oracle Functions. | _**Logging >> Service Connector Hub >> Functions**_ | | | Move logs from OCI to Datadog using Service Connector Hub, Logging (Source), Functions (Target) and Datadog | [sample](./samples/oci-logs-datadog) | | Send SMS messages for logs using Service Connector Hub, Logging (Source), Functions (Target) and Syniverse SMS | [sample](./samples/oci-notification-syniverse) | +| Move Audit, Logs, and Events from OCI to SumoLogic using Service Connector Hub, Logging (Source), Functions (Target) and HTTPS | [sample](./samples/oci-serviceconnector-sumologic-python/) | | | | | _**Streaming >> Service Connector Hub >> Functions**_ | | | Convert JSON to CSV format using Service Connector Hub, Streams (Source and Target) and Functions (Task) | [sample](./samples/oci-serviceconnector-streaming-json-to-csv-python) | diff --git a/samples/oci-serviceconnector-sumologic-python/README.md b/samples/oci-serviceconnector-sumologic-python/README.md new file mode 100644 index 0000000..ceb2991 --- /dev/null +++ b/samples/oci-serviceconnector-sumologic-python/README.md @@ -0,0 +1,116 @@ +# Send OCI Events and Audit to SumoLogic via HTTPS + +This is a sample to use a Service Connector Hub with Function as a target. This function will take a single JSON or list of JSON objects, and write them to a configured SumoLogic endpoint. + +This sample is based on the [documented SumoLogic Connector](https://help.sumologic.com/docs/send-data/collect-from-other-data-sources/collect-logs-oracle-cloud-infrastructure/) in OCI Docs, but with a couple of minor differences: +* It sends JSON documents instead of text +* Debugging is an option + +As you make your way through this tutorial, look out for this icon ![user input icon](./images/userinput.png). +Whenever you see it, it's time for you to perform an action. + + +## Prerequisites + +Before you deploy this sample function, make sure you have run steps A, B +and C of the [Oracle Functions Quick Start Guide for Cloud Shell](https://www.oracle.com/webfolder/technetwork/tutorials/infographics/oci_functions_cloudshell_quickview/functions_quickview_top/functions_quickview/index.html) +* A - Set up your tenancy +* B - Create application +* C - Set up your Cloud Shell dev environment + + +## List Applications + +Assuming you have successfully completed the prerequisites, you should see your +application in the list of applications. + +``` +fn ls apps +``` + + +## Review and customize the function + +Review the following files in the current folder: +* the code of the function, [func.py](./func.py) +* its dependencies, [requirements.txt](./requirements.txt) +* the function metadata, [func.yaml](./func.yaml) + + +### Configure SumoLogic Endpoint + +Within the `func.yaml` file, there is a configuration parameter called `SUMOLOGIC_ENDPOINT`. This must be set either prior to deployment, or after deployment using the UI/CLI. + +See [this link](https://help.sumologic.com/docs/send-data/hosted-collectors/http-source/logs-metrics/upload-logs/) how to generate this URL, and for additional headers and changes that could be made to this function. + +The function here also allows a configuration called `DEBUG` - set this to true if additional output is desired. + + +## Deploy the function + +In Cloud Shell, run the `fn deploy` command to build *this* function and its dependencies as a Docker image, +push the image to the specified Docker registry, and deploy *this* function to Oracle Functions +in the application created earlier: + +![user input icon](./images/userinput.png) +``` +fn -v deploy --app +``` +e.g., +``` +fn -v deploy --app myapp +``` + +### Test + +In Cloud Shell, run the `fn invoke` ccommand to unit test this function: + +![user input icon](./images/userinput.png) +``` +fn invoke < test.json +``` +e.g., +``` +fn invoke myapp oci-serviceconnector-sumologic-python < examples-json/audit.json +``` + +You should see no error or output returned by the function. Function Logs will show a record of the call, and if `DEBUG` was enabled, then more will be shown. + + +## Create a Service Connector for OCI Audit Logging + +1. From the navigation menu, select **Logging**, and then select **Service Connectors**. + +2. Click Create Connector, add a Name, Description, select the compartment, select the Source as **Logging** and Target as **Functions** with no Task. + +3. On Configure Source connection, select the compartment, select the _Audit log with sub-compartments enabled. (Doing so at the tenancy root will send ALL audit to SumoLogic) + +4. On Configure Target connection, select the Functions compartment, the Functions App, and the `sumologicfn` Function. If prompted to create a policy, click Create. + +### Screeen Shots + +Service Connector Source +![Audit Log SCH](./images/ServiceConnectorAuditLog.png) + +Service Connector Target +![Functions Target SCH](./images/ServiceConnectorFunctionTarget.png) + +### OCI Events + +This function can support OCI Events or multiple logs that are not audit. The following screen shot shows a couple OCI Events for Cloud Guard going to the same function: + +![Events to Function](./images/EventsToFunction.png) + +### Logging Filter + +Service Connectors support Log filters as well. In order to limit the logs sent to SumoLogic, the following example filter could be applied, which will only look for POST or PUT events, and prevent the listed message types from being sent at all. + +``` +search "ocid1.compartment.oc1..xxxxx/_Audit_Include_Subcompartment" | (data.request.action='POST' or data.request.action='PUT') and (type!='io.k8s.coordination.v1.leases.update') and (type!='io.k8s.authentication.v1.tokenreviews.create') and (type!='com.oraclecloud.management-agent.ExchangeWorkSubmission') +``` + +## Monitoring Functions and Service Connector + +Make sure you configure basic observability for your function and connector using metrics, alarms and email alerts: +* [Basic Guidance for Monitoring your Functions](../basic-observability/functions.md) +* [Basic Guidance for Monitoring your Service Connector](../basic-observability/service-connector-hub.md) diff --git a/samples/oci-serviceconnector-sumologic-python/examples-json/audit.json b/samples/oci-serviceconnector-sumologic-python/examples-json/audit.json new file mode 100644 index 0000000..c6d5652 --- /dev/null +++ b/samples/oci-serviceconnector-sumologic-python/examples-json/audit.json @@ -0,0 +1,225 @@ +[ + { + "data": { + "additionalDetails": { + "X-Real-Port": 19991 + }, + "availabilityDomain": "AD3", + "compartmentId": "ocid1.compartment.oc1..xxx", + "compartmentName": "Network", + "definedTags": null, + "eventGroupingId": "/xxx", + "eventName": "ListPrivateIps", + "freeformTags": null, + "identity": { + "authType": "instance", + "callerId": null, + "callerName": null, + "consoleSessionId": null, + "credentials": null, + "ipAddress": "xx.xx.xx.xx", + "principalId": "ocid1.instance.oc1.iad.xxx", + "principalName": null, + "tenantId": "ocid1.tenancy.oc1..xxx", + "userAgent": null + }, + "message": "ListPrivateIps succeeded", + "request": { + "action": "GET", + "headers": { + "Accept": [ + "*/*" + ], + "Authorization": [ + "Bearer yyy.zzz.rpVTVki1Fo-aaa" + ], + "Date": [ + "Fri, 17 Mar 2023 12:57:44 GMT" + ] + }, + "id": "/xxx/3ABE1F95B35432E3683550035916DA4D", + "parameters": { + "vnicId": [ + "ocid1.vnic.oc1.iad.xxx" + ] + }, + "path": "/20160918/privateIps" + }, + "resourceId": null, + "response": { + "headers": { + "Connection": [ + "close" + ], + "Content-Length": [ + "1446" + ], + "Content-Type": [ + "application/json" + ], + "Date": [ + "Fri, 17 Mar 2023 12:57:44 GMT" + ], + "X-Content-Type-Options": [ + "nosniff" + ], + "opc-request-id": [ + "/xx/yy" + ] + }, + "message": null, + "payload": {}, + "responseTime": "2023-03-17T12:57:44.914Z", + "status": "200" + }, + "stateChange": { + "current": null, + "previous": null + } + }, + "dataschema": "2.0", + "id": "45f1403b-8909-40df-aa69-769f867a30df", + "oracle": { + "compartmentid": "ocid1.compartment.oc1..xx", + "ingestedtime": "2023-03-17T12:57:45.842Z", + "loggroupid": "_Audit", + "tenantid": "ocid1.tenancy.oc1..xx" + }, + "source": "", + "specversion": "1.0", + "time": "2023-03-17T12:57:44.914Z", + "type": "com.oraclecloud.virtualNetwork.ListPrivateIps" + }, + { + "data": { + "additionalDetails": { + "X-Real-Port": 52346 + }, + "availabilityDomain": "AD3", + "compartmentId": "ocid1.compartment.oc1..xx", + "compartmentName": "Network", + "definedTags": null, + "eventGroupingId": "sch-7v3vazqt2ohujqmxuizhbg7dfcq/xx", + "eventName": "UploadOCSFile", + "freeformTags": null, + "identity": { + "authType": "resource", + "callerId": null, + "callerName": null, + "consoleSessionId": null, + "credentials": "***", + "ipAddress": "10.1.8.240", + "principalId": "ocid1.serviceconnector.oc1.iad.xx", + "principalName": null, + "tenantId": "ocid1.tenancy.oc1..xx", + "userAgent": "Oracle-JavaSDK/2.46.0 (Linux/4.14.35-2047.521.4.el7uek.x86_64; Java/1.8.0_311; Java HotSpot(TM) 64-Bit Server VM GraalVM EE 20.3.4/25.311-b11-jvmci-20.3-b24)" + }, + "message": "UploadOCSFile succeeded", + "request": { + "action": "POST", + "headers": { + "Accept": [ + "application/json" + ], + "Authorization": [ + "Signature ***" + ], + "Connection": [ + "keep-alive" + ], + "Content-Length": [ + "10452526" + ], + "Content-Type": [ + "application/json" + ], + "Date": [ + "Fri, 17 Mar 2023 12:57:42 GMT" + ], + "User-Agent": [ + "Oracle-JavaSDK/2.46.0 (Linux/4.14.35-2047.521.4.el7uek.x86_64; Java/1.8.0_311; Java HotSpot(TM) 64-Bit Server VM GraalVM EE 20.3.4/25.311-b11-jvmci-20.3-b24)" + ], + "opc-client-info": [ + "Oracle-JavaSDK/2.46.0" + ], + "opc-request-id": [ + "sch-7v3vazqt2oujqmxuiwzhbg7dfcq" + ], + "opc-retry-token": [ + "69ab78a0-82a6-4ab-8f63-6c83c775e194" + ], + "x-content-sha256": [ + "NwJ8bFtpyzoEg3lKe8yGsRHkMEkrV7P2UFqowRds8U=" + ] + }, + "id": "sch-7v3vazqt2ohujqmxuiwzhbg7dfcq/yy/zz", + "parameters": { + "logGroupId": [ + "ocid1.loganalyticsloggroup.oc1.iad.xx" + ], + "serviceConnectorCompartmentId": [ + "ocid1.compartment.oc1..xx" + ], + "serviceConnectorId": [ + "ocid1.serviceconnector.oc1.iad.xx" + ], + "serviceConnectorSource": [ + "LOG" + ], + "serviceConnectorSourceId": [ + "null" + ] + }, + "path": "/20200601/actions/uploadOCSFile" + }, + "resourceId": "uploadOCSFile", + "response": { + "headers": { + "Content-Length": [ + "48" + ], + "Content-Type": [ + "application/json" + ], + "Date": [ + "Fri, 17 Mar 2023 12:57:42 GMT" + ], + "X-Content-Type-Options": [ + "nosniff" + ], + "opc-content-md5": [ + "b8ymx7ezMarR/7wFJq/rNg==" + ], + "opc-object-id": [ + "8a04e7ec-95b1-4a30-8e4f-76506f1ce3cc" + ], + "opc-request-id": [ + "sch-xx/xx/yy" + ] + }, + "message": null, + "payload": {}, + "responseTime": "2023-03-17T12:57:43.674Z", + "status": "200" + }, + "stateChange": { + "current": { + "timeCreated": "2023-03-17T12:57:43.000Z" + }, + "previous": {} + } + }, + "dataschema": "2.0", + "id": "82bc9a7c-876c-424b-b732-a6c19ca73dc9", + "oracle": { + "compartmentid": "ocid1.compartment.oc1..xx", + "ingestedtime": "2023-03-17T12:57:53.467Z", + "loggroupid": "_Audit", + "tenantid": "ocid1.tenancy.oc1..xx" + }, + "source": "", + "specversion": "1.0", + "time": "2023-03-17T12:57:43.674Z", + "type": "com.oraclecloud.LoggingAnalytics.UploadOCSFile" + } +] \ No newline at end of file diff --git a/samples/oci-serviceconnector-sumologic-python/examples-json/cloudguard-example.json b/samples/oci-serviceconnector-sumologic-python/examples-json/cloudguard-example.json new file mode 100644 index 0000000..cbe5e85 --- /dev/null +++ b/samples/oci-serviceconnector-sumologic-python/examples-json/cloudguard-example.json @@ -0,0 +1,44 @@ +{ + "eventType": "com.oraclecloud.cloudguard.problemdetected", + "cloudEventsVersion": "0.1", + "eventTypeVersion": "2.0", + "source": "CloudGuardResponderEngine", + "eventTime": "2023-03-21T22:23:22Z", + "contentType": "application/json", + "data": { + "compartmentId": "ocid1.compartment.oc1..xx", + "compartmentName": "CIS235-appdev-cmp", + "resourceName": "Instance terminated", + "resourceId": "ocid1.cloudguardproblem.oc1.iad.xx", + "additionalDetails": { + "tenantId": "ocid1.tenancy.oc1..xx", + "status": "OPEN", + "reason": "New Problem detected by CloudGuard", + "problemName": "INSTANCE_TERMINATED", + "riskLevel": "HIGH", + "problemType": "ACTIVITY", + "resourceName": "a@A.com", + "resourceId": "ocid1.saml2idp.oc1..xx/a@a.com", + "resourceType": "User", + "targetId": "ocid1.cloudguardtarget.oc1.iad.xx", + "labels": "Compute", + "firstDetected": "2023-03-21T22:22:41.812Z", + "lastDetected": "2023-03-21T22:22:41.812Z", + "region": "us-ashburn-1", + "impactedResourceName": "wls-ucm8", + "impactedResourceId": "ocid1.instance.oc1.iad.xx", + "impactedResourceType": "Instance", + "problemAdditionalDetails": { + "imageId": "ocid1.image.oc1..xx", + "shape": "VM.Standard.E4.Flex", + "type": "CustomerVmi" + }, + "problemDescription": "Compute instances may deliver critical functions. ", + "problemRecommendation": "Ensure that the termination of the compute instance is sanctioned and performed by a permitted administrator." + } + }, + "eventID": "7c9615da-209d-4cd5-96b1-c11354709cc3", + "extensions": { + "compartmentId": "ocid1.compartment.oc1..xx" + } +} \ No newline at end of file diff --git a/samples/oci-serviceconnector-sumologic-python/func.py b/samples/oci-serviceconnector-sumologic-python/func.py new file mode 100644 index 0000000..d01804a --- /dev/null +++ b/samples/oci-serviceconnector-sumologic-python/func.py @@ -0,0 +1,58 @@ +import io +import json +import logging +import os + +import requests + +# Handle JSON List and send as messages to SumoLogic +def handler(ctx, data: io.BytesIO = None): + logger = logging.getLogger() + logger.setLevel(logging.INFO) + + # Sumologic endpoint URL to upload OCI logs to HTTP custom app. + # this value will be defined defined in func.yaml + sumologic_endpoint = os.environ['SUMOLOGIC_ENDPOINT'] + debug = True if os.environ['DEBUG'].casefold() == "true" else False + if debug: + logger.setLevel(logging.DEBUG) + logger.debug("Function start") + + # If the following is added, the function returns the request, so that SCH can send to actual target + + try: + logentries = json.loads(data.getvalue()) # deserialize the bytesstream input as JSON array + if not isinstance(logentries, list): + logger.debug(f'JSON entry from Event: {logentries}') + try: + response_from_sumologic = requests.post(sumologic_endpoint, + json=logentries, + headers={}) + logging.getLogger().debug(f"From Sumo: {response_from_sumologic}") + except requests.exceptions.HTTPError as e: + logger.error(f"HTTPError: {e.errno}") + return + else: + # Optional...log the input to the function as human readble JSON. + # Not to be used in production + logger.debug(f"JSON list from SCH - Logs to process: {len(logentries)}") + + for logEntry in logentries: + logger.debug(f"Log Entry JSON: {logEntry}") + # Post the JSON as-is with no headers + # Try and eat error + try: + response_from_sumologic = requests.post(sumologic_endpoint, + json=logEntry, + headers={}) + logging.getLogger().debug(f"From Sumo: {response_from_sumologic}") + except requests.exceptions.HTTPError as e: + logger.error(f"HTTPError: {e.errno}") + + logger.debug("function end - No return") + return + + except Exception as e: + logger.error("Failure in the function: {}".format(str(e))) + return + diff --git a/samples/oci-serviceconnector-sumologic-python/func.yaml b/samples/oci-serviceconnector-sumologic-python/func.yaml new file mode 100644 index 0000000..0625cc5 --- /dev/null +++ b/samples/oci-serviceconnector-sumologic-python/func.yaml @@ -0,0 +1,12 @@ +schema_version: 20180708 +name: sumologicfn +version: 0.0.1 +runtime: python +build_image: fnproject/python:3.9-dev +run_image: fnproject/python:3.9 +entrypoint: /python/bin/fdk /function/func.py handler +memory: 256 +timeout: 30 +config: + DEBUG: "false" + SUMOLOGIC_ENDPOINT: https://endpoint4.collection.sumologic.com/receiver/v1/http/ diff --git a/samples/oci-serviceconnector-sumologic-python/images/EventsToFunction.png b/samples/oci-serviceconnector-sumologic-python/images/EventsToFunction.png new file mode 100644 index 0000000..7fa66ef Binary files /dev/null and b/samples/oci-serviceconnector-sumologic-python/images/EventsToFunction.png differ diff --git a/samples/oci-serviceconnector-sumologic-python/images/ServiceConnectorAuditLog.png b/samples/oci-serviceconnector-sumologic-python/images/ServiceConnectorAuditLog.png new file mode 100644 index 0000000..7d84c07 Binary files /dev/null and b/samples/oci-serviceconnector-sumologic-python/images/ServiceConnectorAuditLog.png differ diff --git a/samples/oci-serviceconnector-sumologic-python/images/ServiceConnectorFunctionTarget.png b/samples/oci-serviceconnector-sumologic-python/images/ServiceConnectorFunctionTarget.png new file mode 100644 index 0000000..bdcde97 Binary files /dev/null and b/samples/oci-serviceconnector-sumologic-python/images/ServiceConnectorFunctionTarget.png differ diff --git a/samples/oci-serviceconnector-sumologic-python/images/userinput.png b/samples/oci-serviceconnector-sumologic-python/images/userinput.png new file mode 100644 index 0000000..ce6a202 Binary files /dev/null and b/samples/oci-serviceconnector-sumologic-python/images/userinput.png differ diff --git a/samples/oci-serviceconnector-sumologic-python/requirements.txt b/samples/oci-serviceconnector-sumologic-python/requirements.txt new file mode 100644 index 0000000..e91de0f --- /dev/null +++ b/samples/oci-serviceconnector-sumologic-python/requirements.txt @@ -0,0 +1,2 @@ +fdk +requests \ No newline at end of file