OTLP export crashes with log message from twisted

[lukaslihotzki-f]

Describe your environment

OS: Arch Linux
Python version: Python 3.13.2
SDK version: opentelemetry-sdk 1.31.0
API version: opentelemetry-api 1.31.0

What happened?

I integrated the OTLP exporter into synapse. When starting a new homeserver, this exception is raised:

Traceback (most recent call last):
  File "venv/lib/python3.13/site-packages/opentelemetry/sdk/_logs/_internal/export/__init__.py", line 308, in _export_batch
    self._exporter.export(self._log_records[:idx])  # type: ignore
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "venv/lib/python3.13/site-packages/opentelemetry/exporter/otlp/proto/http/_log_exporter/__init__.py", line 159, in export
    serialized_data = encode_logs(batch).SerializeToString()
                      ~~~~~~~~~~~^^^^^^^
  File "venv/lib/python3.13/site-packages/opentelemetry/exporter/otlp/proto/common/_internal/_log_encoder/__init__.py", line 37, in encode_logs
    return ExportLogsServiceRequest(resource_logs=_encode_resource_logs(batch))
                                                  ~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "venv/lib/python3.13/site-packages/opentelemetry/exporter/otlp/proto/common/_internal/_log_encoder/__init__.py", line 72, in _encode_resource_logs
    pb2_log = _encode_log(sdk_log)
  File "venv/lib/python3.13/site-packages/opentelemetry/exporter/otlp/proto/common/_internal/_log_encoder/__init__.py", line 58, in _encode_log
    body=_encode_value(body, allow_null=True),
         ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "venv/lib/python3.13/site-packages/opentelemetry/exporter/otlp/proto/common/_internal/__init__.py", line 102, in _encode_value
    raise Exception(f"Invalid type {type(value)} of value {value}")
Exception: Invalid type <class 'twisted.logger._stdlib.StringifiableFromEvent'> of value Redirected stdout/stderr to logs

Steps to Reproduce

Use the STDLibLogObserver from twisted, which passes a StringifiableFromEvent to the logger, which doesn't provide any interface supported by _encode_value.

# setup OTLP logging
from opentelemetry._logs import set_logger_provider
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.sdk.resources import Resource
import logging

class OtlpHandler(LoggingHandler):
    def __init__(self, level=logging.NOTSET, logger_provider=None) -> None:
        logger_provider = LoggerProvider(resource=Resource(attributes={"service.name": "mini"}))
        logger_provider.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter()))
        super().__init__(level, logger_provider)

logging.getLogger().addHandler(OtlpHandler())


# setup twisted and log through twisted
from twisted.logger import LogBeginner, LogPublisher, STDLibLogObserver, ILogObserver
import sys, warnings

observer = STDLibLogObserver()
logBeginner = LogBeginner(LogPublisher(), sys.stderr, sys, warnings)
logBeginner.beginLoggingTo([observer], redirectStandardIO=True)
print("Hello", file=sys.stderr)

import time
time.sleep(40)

Expected Result

The stringified message body is logged.

Actual Result

Exception is raised, so logging fails for the whole batch.

Additional context

Ideally, exception handling during log record serializing would be per-record where it makes sense, so a failing record does not impact other log records in the same batch. However, this fix wouldn't be sufficient to get the expected result.

Would you like to implement a fix?

Yes

[jomcgi]

I found an old issue when researching the spec / history where the same fix (str fallback) was proposed.
#3389

Some relevant comments:

Not sure if we should be making any changes here, the supported types are defined in the proto.

There should not be other value types besides the ones defined in the proto.

Could this be resolved for the third party library through autoinstrumentation instead of requiring a core exporter to implement something that is outside of the spec?

[xrmx]

@lukaslihotzki-f Could you please give #4528 a try?