OTLP Exporter cannot handle Exceptions

[siebl]

Describe your environment

OS: Windows
Python version: Python 3.12.3
SDK version: 1.31.1
API version: 1.31.1

What happened?

OTLP cannot export, if an exception Object is logged.

Steps to Reproduce

• configure your logger to use and export to a OTEL instance (easiest Docker run grafana/otel-lgtm)
• log any kind of exception object e.g: logger.debug(Exception("This is a Test"))

Expected Result

Exception is logged and exported to OTEL

• Log message is the Exception Message ("This is a Test")
• Stacktrace is available

Actual Result

Logs will contain following entry:
level:
ERROR
message:
"Exception while exporting logs."
exception_stacktrace:

Traceback (most recent call last):
  File "\venv\Lib\site-packages\opentelemetry\sdk\_logs\_internal\export\__init__.py", line 308, in _export_batch
    self._exporter.export(self._log_records[:idx])  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\opentelemetry\exporter\otlp\proto\grpc\_log_exporter\__init__.py", line 111, in expo
rt
    return self._export(batch)
           ^^^^^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\opentelemetry\exporter\otlp\proto\grpc\exporter.py", line 299, in _export
    request=self._translate_data(data),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\opentelemetry\exporter\otlp\proto\grpc\_log_exporter\__init__.py", line 108, in _tra
nslate_data
    return encode_logs(data)
           ^^^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\opentelemetry\exporter\otlp\proto\common\_internal\_log_encoder\__init__.py", line 3
7, in encode_logs
    return ExportLogsServiceRequest(resource_logs=_encode_resource_logs(batch))
                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\opentelemetry\exporter\otlp\proto\common\_internal\_log_encoder\__init__.py", line 7
2, in _encode_resource_logs
    pb2_log = _encode_log(sdk_log)
              ^^^^^^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\opentelemetry\exporter\otlp\proto\common\_internal\_log_encoder\__init__.py", line 5
8, in _encode_log
    body=_encode_value(body, allow_null=True),
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\opentelemetry\exporter\otlp\proto\common\_internal\__init__.py", line 102, in _encod
e_value
    raise Exception(f"Invalid type {type(value)} of value {value}")
Exception: Invalid type <class 'Exception'> of value This is a Test

Additional context

Looking at the stacktrace, my guess would be to extend following function with handling for Exception types:
(opentelemetry/exporter/otlp/proto/common/_internal/__init__.py)

def _encode_value(
    value: Any, allow_null: bool = False
) -> Optional[PB2AnyValue]:
    if allow_null is True and value is None:
        return None
    if isinstance(value, bool):
        return PB2AnyValue(bool_value=value)
    if isinstance(value, str):
        return PB2AnyValue(string_value=value)
    if isinstance(value, int):
        return PB2AnyValue(int_value=value)
    if isinstance(value, float):
        return PB2AnyValue(double_value=value)
    if isinstance(value, bytes):
        return PB2AnyValue(bytes_value=value)
    if isinstance(value, Sequence):
        return PB2AnyValue(
            array_value=PB2ArrayValue(
                values=_encode_array(value, allow_null=allow_null)
            )
        )
    elif isinstance(value, Mapping):
        return PB2AnyValue(
            kvlist_value=PB2KeyValueList(
                values=[
                    _encode_key_value(str(k), v, allow_null=allow_null)
                    for k, v in value.items()
                ]
            )
        )
    raise Exception(f"Invalid type {type(value)} of value {value}")

However, I am completely unfamiliar with the Architecture and Structure of this Project. This is why this ticket and the kind request to enable logging Excepion Objects.

Would you like to implement a fix?

None

[xrmx]

This is a dup of #4509

[siebl]

Thanks @xrmx for linking the issues!

I tested the changes of @lukaslihotzki-f locally and it resolves the bug. However, I don't think it is an ideal solution. Please hear me out considering this example:

import logging
import sys

from opentelemetry._logs import set_logger_provider
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
from opentelemetry.instrumentation.logging import LoggingInstrumentor
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.sdk.resources import Resource
from opentelemetry.semconv.resource import ResourceAttributes

resource = Resource.create({
    ResourceAttributes.SERVICE_NAME: "TEST_LOGGING",
})

logger_provider = LoggerProvider(resource=resource)
set_logger_provider(logger_provider)

# Set up log exporter
log_exporter = OTLPLogExporter(endpoint="http://localhost:4317")
logger_provider.add_log_record_processor(BatchLogRecordProcessor(log_exporter))

logger = logging.getLogger()
logger.setLevel(logging.DEBUG)

# Console handler
console_handler = logging.StreamHandler(sys.stdout)
logger.addHandler(console_handler)

# OTLP handler
otlp_handler = LoggingHandler(logger_provider=logger_provider)
logger.addHandler(otlp_handler)

# Instrument logging
LoggingInstrumentor().instrument(set_logging_format=True)


def do_things():
    raise AssertionError("This is a Test")


try:
    do_things()
except Exception as e:
    logger.info("As expected")
    logger.error(e)
    logger.exception(e)

This will generate two (three actually) log entries:

• logger.error(e) produces
  • just the Line "This is a Test"
• logger.exception(e) produces
  • the Line "This is a Test"
  • populates the field exception_stacktrace with the full Traceback of the Exception

I know, you are supposed to use logger.exception() for logging an exception and therefore i would consider the bug closed. However, if one of my more junior dev's uses the wrong logging function, i have an entry which doesn't give me much information.

My suggestion would be to use traceback.format_tb to encode Exceptions. That way the full stack is available, but it won't look that neat. I am aware that this is opinionated. So please change this issue to a feature-request or discussion topic as you see fit.

[xrmx]

@siebl Please give #4528 a try