|
| 1 | +--- |
| 2 | +title: OpenTelemetry for Generative AI |
| 3 | +linkTitle: OTel for GenAI |
| 4 | +date: 2024-12-05 |
| 5 | +author: >- |
| 6 | + [Drew Robbins](https://github.com/drewby) (Microsoft), [Liudmila |
| 7 | + Molkova](https://github.com/lmolkova) (Microsoft) |
| 8 | +issue: https://github.com/open-telemetry/opentelemetry.io/issues/5581 |
| 9 | +sig: SIG GenAI Observability |
| 10 | +cSpell:ignore: genai liudmila molkova |
| 11 | +--- |
| 12 | + |
| 13 | +As organizations increasingly adopt Large Language Models (LLMs) and other |
| 14 | +generative AI technologies, ensuring reliable performance, efficiency, and |
| 15 | +safety is essential to meet user expectations, optimize resource costs, and |
| 16 | +safeguard against unintended outputs. Effective observability for AI operations, |
| 17 | +behaviors, and outcomes can help meet these goals. OpenTelemetry is being |
| 18 | +enhanced to support these needs specifically for generative AI. |
| 19 | + |
| 20 | +Two primary assets are in development to make this possible: **Semantic |
| 21 | +Conventions** and **Instrumentation Libraries**. The first instrumentation |
| 22 | +library targets the |
| 23 | +[OpenAI Python API library](https://pypi.org/project/openai/). |
| 24 | + |
| 25 | +[**Semantic Conventions**](/docs/concepts/semantic-conventions/) establish |
| 26 | +standardized guidelines for how telemetry data is structured and collected |
| 27 | +across platforms, defining inputs, outputs, and operational details. For |
| 28 | +generative AI, these conventions streamline monitoring, troubleshooting, and |
| 29 | +optimizing AI models by standardizing attributes such as model parameters, |
| 30 | +response metadata, and token usage. This consistency supports better |
| 31 | +observability across tools, environments, and APIs, helping organizations track |
| 32 | +performance, cost, and safety with ease. |
| 33 | + |
| 34 | +The |
| 35 | +[**Instrumentation Library**](/docs/specs/otel/overview/#instrumentation-libraries) |
| 36 | +is being developed within the |
| 37 | +[OpenTelemetry Python Contrib](https://github.com/open-telemetry/opentelemetry-python-contrib) |
| 38 | +under |
| 39 | +[instrumentation-genai](https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation-genai) |
| 40 | +project to automate telemetry collection for generative AI applications. The |
| 41 | +first release is a Python library for instrumenting OpenAI client calls. This |
| 42 | +library captures spans and events, gathering essential data like model inputs, |
| 43 | +response metadata, and token usage in a structured format. |
| 44 | + |
| 45 | +## Key Signals for Generative AI |
| 46 | + |
| 47 | +The [Semantic Conventions for Generative AI](/docs/specs/semconv/gen-ai/) focus |
| 48 | +on capturing insights into AI model behavior through three primary signals: |
| 49 | +[Traces](/docs/concepts/signals/traces/), |
| 50 | +[Metrics](/docs/concepts/signals/metrics/), and |
| 51 | +[Events](/docs/specs/otel/logs/event-api/). |
| 52 | + |
| 53 | +Together, these signals provide a comprehensive monitoring framework, enabling |
| 54 | +better cost management, performance tuning, and request tracing. |
| 55 | + |
| 56 | +### Traces: Tracing Model Interactions |
| 57 | + |
| 58 | +Traces track each model interaction's lifecycle, covering input parameters (for |
| 59 | +example, temperature, top_p) and response details like token count or errors. |
| 60 | +They provide visibility into each request, aiding in identifying bottlenecks and |
| 61 | +analyzing the impact of settings on model output. |
| 62 | + |
| 63 | +### Metrics: Monitoring Usage and Performance |
| 64 | + |
| 65 | +Metrics aggregate high-level indicators like request volume, latency, and token |
| 66 | +counts, essential for managing costs and performance. This data is particularly |
| 67 | +critical for API-dependent AI applications with rate limits and cost |
| 68 | +considerations. |
| 69 | + |
| 70 | +### Events: Capturing Detailed Interactions |
| 71 | + |
| 72 | +Events log detailed moments during model execution, such as user prompts and |
| 73 | +model responses, providing a granular view of model interactions. These insights |
| 74 | +are invaluable for debugging and optimizing AI applications where unexpected |
| 75 | +behaviors may arise. |
| 76 | + |
| 77 | +{{% alert title="Note" color="info" %}} Note that we decided to use |
| 78 | +[events emitted](/docs/specs/otel/logs/api/#emit-an-event) with the |
| 79 | +[Logs API](/docs/specs/otel/logs/api/) specification in the Semantic Conventions |
| 80 | +for Generative AI. Events allows for us to define specific |
| 81 | +[semantic conventions](/docs/specs/semconv/general/events/) for the user prompts |
| 82 | +and model responses that we capture. This addition to the API is in development |
| 83 | +and considered unstable.{{% /alert %}} |
| 84 | + |
| 85 | +### Extending Observability with Vendor-Specific Attributes |
| 86 | + |
| 87 | +The Semantic Conventions also define vendor-specific attributes for platforms |
| 88 | +like OpenAI and Azure Inference API, ensuring telemetry captures both general |
| 89 | +and provider-specific details. This added flexibility supports multi-platform |
| 90 | +monitoring and in-depth insights. |
| 91 | + |
| 92 | +## Building the Python Instrumentation Library for OpenAI |
| 93 | + |
| 94 | +This Python-based library for OpenTelemetry captures key telemetry signals for |
| 95 | +OpenAI models, providing developers with an out-of-the-box observability |
| 96 | +solution tailored to AI workloads. The library, |
| 97 | +[hosted within the OpenTelemetry Python Contrib repository](https://github.com/open-telemetry/opentelemetry-python-contrib/tree/opentelemetry-instrumentation-openai-v2%3D%3D2.0b0/instrumentation-genai/opentelemetry-instrumentation-openai-v2), |
| 98 | +automatically collects telemetry from OpenAI model interactions, including |
| 99 | +request and response metadata and token usage. |
| 100 | + |
| 101 | +As generative AI applications grow, additional instrumentation libraries for |
| 102 | +other languages will follow, extending OpenTelemetry support across more tools |
| 103 | +and environments. The current library's focus on OpenAI highlights its |
| 104 | +popularity and demand within AI development, making it a valuable initial |
| 105 | +implementation. |
| 106 | + |
| 107 | +### Example Usage |
| 108 | + |
| 109 | +Here's an example of using the OpenTelemetry Python library to monitor a |
| 110 | +generative AI application with the OpenAI client. |
| 111 | + |
| 112 | +Install the OpenTelemetry dependencies: |
| 113 | + |
| 114 | +```shell |
| 115 | +pip install opentelemetry-distro |
| 116 | +opentelemetry-bootstrap -a install |
| 117 | +``` |
| 118 | + |
| 119 | +Set the following environment variables, updating the endpoint and protocol as |
| 120 | +appropriate: |
| 121 | + |
| 122 | +```shell |
| 123 | +OPENAI_API_KEY=<replace_with_your_openai_api_key> |
| 124 | + |
| 125 | +OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 |
| 126 | +OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf |
| 127 | +OTEL_SERVICE_NAME=python-opentelemetry-openai |
| 128 | +OTEL_LOGS_EXPORTER=otlp_proto_http |
| 129 | +OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true |
| 130 | +# Set to false or remove to disable log events |
| 131 | +OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true |
| 132 | +``` |
| 133 | + |
| 134 | +Then include the following code in your Python application: |
| 135 | + |
| 136 | +```python |
| 137 | +import os |
| 138 | +from openai import OpenAI |
| 139 | + |
| 140 | +client = OpenAI() |
| 141 | +chat_completion = client.chat.completions.create( |
| 142 | + model=os.getenv("CHAT_MODEL", "gpt-4o-mini"), |
| 143 | + messages=[ |
| 144 | + { |
| 145 | + "role": "user", |
| 146 | + "content": "Write a short poem on OpenTelemetry.", |
| 147 | + }, |
| 148 | + ], |
| 149 | +) |
| 150 | +print(chat_completion.choices[0].message.content) |
| 151 | +``` |
| 152 | + |
| 153 | +And then run the example using `opentelemetry-instrument`: |
| 154 | + |
| 155 | +```shell |
| 156 | +opentelemetry-instrument python main.py |
| 157 | +``` |
| 158 | + |
| 159 | +If you do not have a service running to collect telemetry, you can export to the |
| 160 | +console using the following: |
| 161 | + |
| 162 | +```shell |
| 163 | +opentelemetry-instrument --traces_exporter console --metrics_exporter console python main.py |
| 164 | +``` |
| 165 | + |
| 166 | +There is a complete example |
| 167 | +[available here](https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation-genai/opentelemetry-instrumentation-openai-v2/example). |
| 168 | + |
| 169 | +With this simple instrumentation, one can begin capture traces from their |
| 170 | +generative AI application. Here is an example from the |
| 171 | +[Aspire Dashboard](https://learn.microsoft.com/dotnet/aspire/fundamentals/dashboard/standalone?tabs=bash) |
| 172 | +for local debugging. |
| 173 | + |
| 174 | +To start Jaeger, run the following `docker` command and open your web browser |
| 175 | +the `localhost:18888`: |
| 176 | + |
| 177 | +```shell |
| 178 | +docker run --rm -it -d -p 18888:18888 -p 4317:18889 -p 4318:18890 --name aspire-dashboard mcr.microsoft.com/dotnet/aspire-dashboard:9.0 |
| 179 | +``` |
| 180 | + |
| 181 | + |
| 182 | + |
| 183 | +Here is a similar trace captured in |
| 184 | +[Jaeger](https://www.jaegertracing.io/docs/1.63/getting-started/#all-in-one). |
| 185 | + |
| 186 | +To start Jaeger, run the following `docker` command and open your web browser |
| 187 | +the `localhost:16686`. |
| 188 | + |
| 189 | +```shell |
| 190 | +docker run --rm -it -d -p 16686:16686 -p 4317:4317 -p 4318:4318 --name jaeger jaegertracing/all-in-one:latest |
| 191 | +``` |
| 192 | + |
| 193 | + |
| 194 | + |
| 195 | +It's also easy to capture the content history of the chat for debugging and |
| 196 | +improving your application. Simply set the environment variable |
| 197 | +`OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` as follows: |
| 198 | + |
| 199 | +```shell |
| 200 | +export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=True |
| 201 | +``` |
| 202 | + |
| 203 | +This will turn on content capture which collects OpenTelemetry events containing |
| 204 | +the payload: |
| 205 | + |
| 206 | + |
| 207 | + |
| 208 | +## Join Us in Shaping the Future of Generative AI Observability |
| 209 | + |
| 210 | +Community collaboration is key to OpenTelemetry's success. We invite developers, |
| 211 | +AI practitioners, and organizations to contribute, share feedback, or |
| 212 | +participate in discussions. Explore the OpenTelemetry Python Contrib project, |
| 213 | +contribute code, or help shape observability for AI as it continues to evolve. |
| 214 | + |
| 215 | +We now have contributors from [Amazon](https://aws.amazon.com/), |
| 216 | +[Elastic](https://www.elastic.co/), [Google](https://www.google.com/), |
| 217 | +[IBM](https://www.ibm.com), [Langtrace](https://www.langtrace.ai/), |
| 218 | +[Microsoft](https://www.microsoft.com/), [OpenLIT](https://openlit.io/), |
| 219 | +[Scorecard](https://www.scorecard.io/), [Traceloop](https://www.traceloop.com/), |
| 220 | +and more! |
| 221 | + |
| 222 | +You are welcome to join the community! More information can be found at the |
| 223 | +[Generative AI Observability project page](https://github.com/open-telemetry/community/blob/main/projects/gen-ai.md). |
0 commit comments