User defined log config

Author: Starefossen

docs/observability/README.md

@@ -67,17 +67,14 @@ graph LR
 
 Logs are a way to understand what is happening in your application. They are usually text-based and are often used for debugging. Since the format of logs is usually not standardized, it can be difficult to query and aggregate logs and thus we recommend using metrics for dashboards and alerting.
 
-Logs are collected automatically by [fluentd][fluentd], stored in [Elasticsearch][elasticsearch] and made accessible via [Kibana][kibana].
-
-[fluentd]: https://www.fluentd.org/
-[elasticsearch]: https://www.elastic.co/elasticsearch/
-[kibana]: https://www.elastic.co/kibana/
+Logs that are sent to console (`stdout`) are collected automatically and can be configured for persistent storage and querying in several ways.
 
 ```mermaid
 graph LR
-  Application --stdout/stderr--> Fluentbit
-  Fluentbit --> Elasticsearch
-  Elasticsearch --> Kibana
+  Application --stdout/stderr--> Router
+  Router --> A[Secure Logs]
+  Router --> B[Grafana Loki]
+  Router --> C[Elastic / Kibana]
 ```
 
 [:octicons-arrow-right-24: Configure your logs](./logs/README.md)

docs/observability/logs/README.md

@@ -1,18 +1,76 @@
-# Logs
+# Application Logs
 
-## Logging
+## Purpose
 
-Configure your application to log to console \(stdout/stderr\), it will be scraped by [FluentD](https://www.fluentd.org/) running inside the cluster and sent to [Elasticsearch](https://www.elastic.co/products/elasticsearch) and made available via [Kibana](https://www.elastic.co/products/kibana). Visit our Kibana at [logs.adeo.no](https://logs.adeo.no/).
+Logs are a way to understand what is happening in your application. They are usually text-based and are often used for debugging. Since the format of logs is usually not standardized, it can be difficult to query and aggregate logs and thus we recommend using metrics for dashboards and alerting.
 
-If you want more information than just the log message \(loglevel, MDC, etc\), you should log in JSON format; the fields you provide will then be indexed.
+There are many types of logs, and they can be used for different purposes. Some logs are used for debugging, some are used for auditing, and some are used for security. Our primary use case for logs is to understand the flow of a request through a system.
 
-## Working with Kibana
+Application logs in nais is first and foremost a tool for developers to debug their applications. It is not intended to be used for auditing or security purposes. We do not condone writing sensitive information to application logs.
+
+## Overview
+
+Logs that are sent to console (also known as `stdout`) are collected automatically by an agent inside the cluster. This agent can be configured to send logs for persistent storage and querying based on team- and application specific configuration.
+
+Logs must be written in JSON format *and* preferably following the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) or [OpenTelemetry Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/).
+
+```mermaid
+graph LR
+  Application --stdout/stderr--> FluentD
+  FluentD --> A[Secure Logs]
+  FluentD --> B[Grafana Loki]
+  FluentD --> C[Elastic / Kibana]
+```
+
+[FluentD]: https://www.fluentd.org/
+
+## Configuration
+
+
+
+## Destinations
+
+| Destination                 | ID            | Description                        | Availability |
+| :-------------------------- | :------------ | :--------------------------------- | :----------- |
+| None                        | `none`        | Logs are not collected.            | All          |
+| [Loki](#grafana-loki)       | `loki`        | Logs are available in Grafana.     | All          |
+| [Elastic](#elastic-kibana)  | `elastic`     | Logs are available in Kibana.      | NAV-only     |
+| [Secure Logs](#secure-logs) | `secure_logs` | Logs are stored in a secret place. | NAV-only     |
+
+## Grafana Loki
+
+Grafana Loki is a log aggregation system inspired by Prometheus and integrated with Grafana. It is designed to be cost effective and easy to operate, as it does not index the contents of the logs, but rather a set of predefined labels for each log stream.
+
+Grafana Loki is designed to be used in conjunction with metrics and tracing to provide a complete picture of an application's performance. Without the other two, it can be perceived as more cumbersome to use than a traditional logging system.
+
+### Enabling Loki
+
+Loki can be enabled by setting the list of logging destinations in your nais application manifest.
+
+```yaml
+…
+spec:
+  observability:
+    logging:
+      destinations:
+        - id: loki
+```
+
+## Elastic Kibana
+
+Elasticsearch is a
+
+### Configuring Kibana
+
+
+
+### Working with Kibana
 
 When you open Kibana you are prompted to select a workspace, select "Nav Logs" to start viewing your application logs.
 
 Once the page loads you will see an empty page with a search bar. This is the query bar, and it is used to search for logs. You can use the query bar to search for logs by message, by field, or by a combination of both.
 
-The query language is called [Kibana Query Language](https://www.elastic.co/guide/en/kibana/current/kuery-query.html) \(KQL\). KQL is a simplified version of Lucene query syntax. You can use KQL to search for logs by message, by field, or by a combination of both.
+The query language is called [Kibana Query Language](https://www.elastic.co/guide/en/kibana/current/kuery-query.html) (`KQL`). KQL is a simplified version of Lucene query syntax. You can use KQL to search for logs by message, by field, or by a combination of both.
 
 There is also a time picker in the upper right corner of the page. You can use the time picker to select a time range to search for logs. The default time range is the last 15 minutes. If no logs shows up, try to increase the time range.
 
@@ -42,8 +100,7 @@ The following fields are common to all logs and can be used in the query bar:
 | `message: "my message" AND level: "ERROR" AND NOT level: "WARN"` | Search for logs with the message "my message" and the level "ERROR" and not the level "WARN" |
 | `message: "my message" AND level: "ERROR" OR level: "WARN"`      | Search for logs with the message "my message" and the level "ERROR" or the level "WARN"      |
 
-
-## Gain access to logs.adeo.no
+### Gain access to Kibana
 
 In order to get access to logs.adeo.no you need to have the correct access rights added to your AD account. This can be requested through your Personnal Manager.
 
@@ -66,7 +123,7 @@ Secure logs can be enabled by setting the `secureLogs.enabled` flag in the appli
 
 ### Log files
 
-With secure logs enabled a directory `/secure-logs/` will be mounted in the application container. Every `*.log` file in this directory will be monitored and the content transferred to Elasticsearch. Make sure that these files are readable for the log shipper \(the process runs as uid/gid 1065\).
+With secure logs enabled a directory `/secure-logs/` will be mounted in the application container. Every `*.log` file in this directory will be monitored and the content transferred to Elasticsearch. Make sure that these files are readable for the log shipper (`the process runs as uid/gid 1065`).
 
 The `/secure-logs/` directory has a size limit of 128Mb, and it's the application responsibility to ensure that this limit is not exceeded. **If the limit is exceeded the application pod will be evicted and restarted.** Use log rotation on file size to avoid this.
 
@@ -111,29 +168,20 @@ If you do not want to have these logs as files in the pod, it is also possible t
 curl -X POST -d '{"log":"hello world","field1":"value1"}' -H 'Content-Type: application/json' http://localhost:19880/
 ```
 
-## Audit logs
-
-Most applications where a user processes data related to another user need to log audit statements, detailing which user did what action on which subject.
-These logs need to follow a specific format and be accessible by ArcSight.
-See [naudit](https://github.com/navikt/naudit) for how to set up the logging, and details on the log format.
-
-## Overview
-
-![The flow diagram shows that you can configure your app to log to a file using stdout or stderr, run FluentD inside the cluster to scrape the logs and send it to Elasticsearch. This will make the logs available via Kibana.](../../assets/logging_overview.png)
-
-## Gaining access in kibana
+### Gain access to Secure Logs
 
 Once everything is configured, your secure logs will be sent to the `tjenestekall-*` index in kibana. To gain access to these logs, you need to do the following:
 
-### 1 Create an AD-group
+#### 1 Create an AD-group
 
 To make sure you gain access to the proper logs, you need an AD-group connected to the nais-team. So the first thing you do is create this group.
 
 Go to [Porten (service desk)](https://jira.adeo.no/plugins/servlet/desk/portal/542) and click `Melde sak til IT`. The follow the template below.
 For IT to be able to correctly add the group to Remedy you need to specify the four digit department code for those who can be able to ask for permission to the group. E.g 2990 is the four digit code for the department IT-AVDELINGEN. If you are creating secure logs for your team and are unsure about which department your colleagues belong to then you can use [Delve](https://eur.delve.office.com/) to search for their profile. In their profile their department code will also be visible.
 
 You can paste the template below into Jira:
-```
+
+```text
 Ønsker å få opprettet en AD-gruppe for å få tilgang til sikker logg i Kibana for applikasjoner knyttet til <your project here>.
 
 Gruppenavn: 0000-GA-SECURE_LOG_<SOMETHING>
@@ -148,20 +196,26 @@ Enheter i Nav som skal ha tilgang: <four digit department code>. E.g (2990 - IT-
 
 ![ticket](../../assets/jira_secure_log.png)
 
-### 2 Connect the AD group to your team in Kibana
+#### 2 Connect the AD group to your team in Kibana
 
 The logs your apps produces are linked with your [nais-team](../../basics/teams.md).
 Administrators of Kibana will create a role for your team with read rights to those logs.
 Whoever is in the AD-group (created in step 1) will get the Kibana role, and can thus read all logs produced by apps belonging to the nais-team.
 
 Ask in the [#atom](https://nav-it.slack.com/archives/C7TQ25L9J) Slack channel to connect the AD-group (created in step 1) to your nais-team.
 
-### 3 Put people into the AD-group
+#### 3 Put people into the AD-group
 
 This must be done by "identansvarlig". For NAV-IT employees, this is `[email protected]`. Send them an email and ask for access with a CC to whoever is your superior.
 
 For everyone else, the team lead or who ever is their superior should know.
 
-### What can go wrong?
+#### What can go wrong?
 
 Basically, the one thing that can go wrong here is that the AD-group is not registered in "identrutinen". If this happens, the group cannot be found by "identansvarlig". If this happens, make a new JIRA-ticket to the same people and tell them to transfer the group. Sadly this can take a few days.
+
+## Audit logs
+
+Most applications where a user processes data related to another user need to log audit statements, detailing which user did what action on which subject.
+These logs need to follow a specific format and be accessible by ArcSight.
+See [naudit](https://github.com/navikt/naudit) for how to set up the logging, and details on the log format.