Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenTelemetry in nais #612

Merged
merged 15 commits into from
Mar 4, 2024
Prev Previous commit
Next Next commit
Add traceql reference
  • Loading branch information
Starefossen committed Mar 4, 2024
commit d47d7ceee54f214b6207956b309cc193cd9ed8b7
14 changes: 12 additions & 2 deletions docs/how-to-guides/observability/tracing/tempo.md
Original file line number Diff line number Diff line change
@@ -14,7 +14,7 @@ Once you have traces being collected, you can visualize and query them in Grafan

The easiest way to get started with querying traces in Grafana Tempo is to use the query builder mode. The query builder mode is a graphical interface that helps you build LogQL queries by selecting labels and fields from your logs.

Start by selecting the `*-tempo` data source for the environment you want to query traces for. Then select the Search query type to open the query builder mode.
Start by selecting the tempo data source for the environment you want to query traces for (one ending with `-tempo`). Then select the Search query type to open the query builder mode.

![Grafana Tempo Query Builder](../../../assets/grafana-tempo-query-builder.png)

@@ -24,7 +24,17 @@ Bellow the query builder you will see the TraceQL query that is being built as y

Click the `Run query` button to run the query and see the results. You can also add the query to a dashboard by clicking the `Add to dashboard` button.

[:octicons-link-external-24: Learn more about Grafana Tempo query editor](https://grafana.com/docs/grafana/latest/datasources/tempo/query-editor/)
[:octicons-link-external-24: Learn more about Grafana Tempo query editor on grafana.com](https://grafana.com/docs/grafana/latest/datasources/tempo/query-editor/)

### TraceQL query language

Grafana Tempo uses the TraceQL query language to query traces. TraceQL is a query language for querying trace data, and it is based on the LogQL query language used by Grafana Loki for querying logs and the PromQL query language used by Prometheus for querying metrics.

TraceQL provides a powerful and flexible way to query trace data, and it is designed to be easy to use and understand. You can use TraceQL to filter and aggregate trace data, and to create visualizations and alerts based on trace data.

<iframe title="vimeo-player" src="https://player.vimeo.com/video/796408188?h=b3e8e3257b" width="100%" height="360" frameborder="0" allowfullscreen></iframe>

[:bulb: Learn more about TraceQL query language](../../../reference/observability/tracing/traceql.md)

## Understanding trace data

18 changes: 9 additions & 9 deletions docs/reference/glossary.md
Original file line number Diff line number Diff line change
@@ -2,37 +2,37 @@

## Observability

Observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs. It is a measure of how well a system's internal states can be inferred from knowledge of its external outputs. In other words, observability is the ability to understand the internal state of a system by examining its outputs.
Observability is the art of understanding how a system behaves by adding instrumentation such as logs, metrics, and traces.

### Metrics

Metrics are a numerical measurement of something in your application. They are useful for understanding the performance of your application and is generally more scalable than logs both in terms of storage and querying since they are structured data.
Metrics are a numerical measurement of something in your application such as the number of requests or the response time. Metrics are much better suited for for dashboards and alerts compared to logs.

### Prometheus

[Prometheus](https://prometheus.io/) is a time-series database that is used to store metrics. It is a very powerful tool that can be used to create alerts and dashboards. Prometheus is used by many open source projects and is the de facto standard for metrics in the cloud native world.
[Prometheus](https://prometheus.io/) is a time-series database that is used to store and query metrics from Grafana.

### Alertmanager

[Alertmanager](https://prometheus.io/docs/alerting/alertmanager/) is a component of Prometheus that is used to create and manage alerts. It can send alerts to various different channels like email, Slack, PagerDuty, etc.
[Alertmanager](https://prometheus.io/docs/alerting/alertmanager/) is a component of Prometheus that is used to create and manage Slack alerts based on the metrics collected by Prometheus.

### Grafana

[Grafana](https://grafana.com/) is a tool for creating dashboards and visualizing data. It is often used in combination with Prometheus to create dashboards and alerts.
[Grafana](https://grafana.com/) is a tool for creating application dashboards and visualizing data such as metrics, traces, and logs in a user-friendly way.

### Traces

Traces are a record of the path a request takes through your application. They are useful for understanding how a request is processed in your application.
Traces are a record of the path a request takes through your application. They are useful for understanding how a request is processed across multiple internal and external services.

### Span

A span represents a single unit of work in a trace, like a SQL query or an HTTP call to an external service.
A span represents a single unit of work in a Trace, like a SQL query or an HTTP call to an API.

### Context

Each Span carries a Context that includes metadata about the trace (like a unique trace identifier and span identifier) and any other data you choose to include. This context is propagated across process boundaries, allowing all the work that's part of a single trace to be linked together, even if it spans multiple services.

### Grafa Tempo
### Grafana Tempo

[Grafana Tempo](https://grafana.com/oss/tempo/) is an open-source, easy-to-use, high-scale, and cost-effective distributed tracing backend that stores and queries traces in a way that is easy to understand and use. It is fully integrated with Grafana, allowing you to visualize and query traces in the same interface as your metrics, and logs.

@@ -44,7 +44,7 @@ Each Span carries a Context that includes metadata about the trace (like a uniqu

Logs are a record of what has happened in your application. They are useful for debugging, but due to their unstructured format they generally do not scale very well.

### Kiibana
### Kibana

[Kibana](https://www.elastic.co/kibana) is a tool for visualizing logs. It is often used in combination with Elasticsearch to create dashboards and alerts.

4 changes: 1 addition & 3 deletions docs/reference/observability/logs/logql.md
Original file line number Diff line number Diff line change
@@ -4,12 +4,10 @@ tags: [reference, loki]
---
# LogQL Reference

[LogQL][logql] is the query language used in Grafana Loki to query logs. It is a powerful query language that allows you to filter, aggregate, and search for logs and should be familiar to anyone who has used SQL or [PromQL](../metrics/promql.md).
LogQL is the query language used in Grafana Loki to query logs. It is a powerful query language that allows you to filter, aggregate, and search for logs and should be familiar to anyone who has used SQL or [PromQL](../metrics/promql.md).

Where LogQL differs from PromQL is it's trailing pipeline syntax, or log pipeline. A log pipeline is a set of stage expressions that are chained together and applied to the selected log streams. Each expression can filter out, parse, or mutate log lines and their respective labels.

[logql]: https://grafana.com/docs/loki/latest/query/

## Syntax

A LogQL query is composed by two main parts: the **stream selector** (the query) and the **log pipeline** (the transformation).
88 changes: 87 additions & 1 deletion docs/reference/observability/tracing/traceql.md
Original file line number Diff line number Diff line change
@@ -4,4 +4,90 @@ tags: [reference, tempo]
---
# TraceQL Reference

* https://grafana.com/docs/loki/latest/query/
TraceQL is the query language used in Grafana Tempo to query traces. It is a powerful query language that allows you to filter, aggregate, and search for traces and should be familiar to anyone who has used SQL or [PromQL](../metrics/promql.md).

Where TraceQL differs from PromQL is it's trailing pipeline syntax, or trace pipeline. A trace pipeline is a set of stage expressions that are chained together and applied to the selected trace data. Each expression can filter out, parse, or mutate trace spans and their respective labels.

## Syntax

A TraceQL query is composed by two main parts: the **trace span selector(s)** (the query) and the **trace pipeline** (aggregations).

```traceql
{label="value"} | stage1 | stage2 | stage3
```

### Trace Span Selector

The trace span selector is used to select spans based on their attributes. It is a set of key-value pairs that are used to filter spans.

Some span metadata are intrinsic to the span, such as `name`, `status`, `duration`, and `kind`, while others (attributes and resources) are user-defined, such as `service.name`, `db.operation`, and `http.status_code`.

```traceql
{ span.http.status_code >= 200 && span.http.status_code < 300 }
```

#### Comparison Operators

Similar to PromQL, TraceQL supports a set of operators for comparing span attributes and values. One notable difference is type coercion, where the type of the attribute is inferred from the value being compared.

- `=` - Equality
- `!=` - Inequality
- `>` - Greater than
- `>=` - Greater than or equal to
- `<` - Less than
- `<=` - Less than or equal to
- `=~` - Regular expression
- `!~` - Negated regular expression

#### Combining spansets

Since a trace can be composed of multiple spans, multiple selectors can be used together to filter spans based on different attributes.

TraceQL supports two types of combining spansets: logical (`&&` and `||`) and structural relations (`>`, `>>`, `<<` `<`, and `~`).

##### Logical

The logical operators `&&` and `||` are used to combine spansets based on their attributes.

```traceql
{ resource.service.name="server" } && { resource.service.name="client" }
```

The above query will return traces where a span with the service name `server` and a differ3ent span with the service name `client` are present.

##### Structural

Structural relations are used to filter spans based on their structural relationships. The structural relations are:

- `>` - Direct parent of
- `>>` - Ancestor of
- `<<` - Descendant of
- `<` - Direct child of
- `~` - Sibling of

For example, to find a trace where a specific HTTP request interacted with a particular kafka topic, you could use the following query:

```traceql
{ span.http.url = "/path/of/api" } >> { span.messaging.destination.name = "my-team.some-topic" }
```

### Trace Pipeline Aggregations

To further refine the selected spans, a trace pipeline can be used to apply a set of aggregation functions on the selected spans.

- `count` - The count of spans in the spanset.
- `avg` - The average of a given numeric attribute or intrinsic for a spanset.
- `max` - The max value of a given numeric attribute or intrinsic for a spanset.
- `min` - The min value of a given numeric attribute or intrinsic for a spanset.
- `sum` - The sum value of a given numeric attribute or intrinsic for a spanset.

We can use the `count()` function to count the number of spans in the selected traces. In the following example, we select traces that contains more then 3 database SELECT operations:

```traceql
{span.db.operation="SELECT"} | count() > 3
```

## Reference

- [TraceQL query documentation](https://grafana.com/docs/tempo/latest/traceql/)
- [Get to know TraceQL](https://grafana.com/blog/2023/02/07/get-to-know-traceql-a-powerful-new-query-language-for-distributed-tracing/)