Under Traces only originating service logs are shown in Elasticsearch Kibana

[AyushCarb1]

Hi Team,

We are using Opentelemetry agent (version: 1.27.0) in our services to emit logs, spans and metrics (auto-instrumentation).
There are in total 3 services involved and all of them are OTEL instrumented . Where the originating services makes a call to another service and that service internally calls another service .
We have OTEL collector which exports data to Elasticsearch APM .
Everything is deployed on AWS EKS .

Observation :

In Kibana, under traces sample we do not see the logs for all the services except that of the originating service .
Trace Sample :

Logs:

This environment variable is set in all the services involved :
OTEL_LOGS_EXPORTER=otlp
Reference : https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk-extensions/autoconfigure/README.md#otlp-exporter-span-metric-and-log-exporters

Questions:
Why don't we see the logs with each Trace for all the services ?
Can you please suggest if there are certain configurations involved to have the logs corresponding to each trace sample for all the spans ?

Note : even using Elasticsearch APM as the otel exporter end point we are observing the same.

[jkwatson]

Hard to guess without you providing how you've configured your otel java agent(s). Can you provide the full set of otel environment variables that you're setting, for each of the services?

[AyushCarb1]

Hi ,
Please, find the environment variables set for all the services :

• name: OTEL_RESOURCE_ATTRIBUTES
  value: service.name=xxxxx-pod
• name: OTEL_TRACES_EXPORTER
  value: otlp
• name: OTEL_EXPORTER_OTLP_ENDPOINT
  value: xxxxx
• name: OTEL_EXPORTER_OTLP_HEADERS
  value: Authorization=Bearer xxxx
• name: OTEL_LOGS_EXPORTER
  value: otlp
• name: OTEL_METRICS_EXPORTER
  value: otlp

The configurations are common for all the services.

[jkwatson]

How are you generating logs, and is there a log bridge configured? I think this might be an instrumentation/agent issue, rather than the core libraries.

[AyushCarb1]

Hi @jkwatson ,

We are using log4j2 as the logging library. The logging configurations are read through log4j2.propeties file .

Please, find the attached property file for log4j2.
`# Declare loggers
packages=com.test.cloud.service.metrics
name=LoggingConfig
monitorInterval=60
appenders=RollingAppender, MetricsAppender

#root logger
rootLogger.level=INFO
rootLogger.appenderRefs=ar_rolling, MetricsAppenderRef
rootLogger.appenderRef.ar_rolling.ref=CatalinaRollingAppender

Console Appender settings

appender.stdout.type=Console
appender.stdout.name=STDOUT
appender.stdout.target=SYSTEM_OUT
appender.stdout.layout.type=PatternLayout
appender.stdout.layout.pattern=%d %p [%t] [%c] [reqid:%X{XILC_req_id} app:%X{webappCtx} jsid:%X{jSessionId} aid:%X{XILC_aid} uid:%X{XILC_uid} org:%X{XILC_org} vorg:%X{XILC_vorg} sid:%X{XILC_sid} un:%X{XILC_un} sn:%X{XILC_sn}] %notEmpty{[jobName:%X{jobName}]}%notEmpty{[correlationId:%X{correlationId}]}%notEmpty{[jobHashCode:%X{jobHashCode}]} - %enc{%m}{CRLF}%n

Rolling Appender settings

Following logic replaces DailyMaxRollingFileAppender

appender.RollingAppender.type=RollingFile
appender.RollingAppender.name=CatalinaRollingAppender
appender.RollingAppender.append=true
appender.RollingAppender.fileName=${sys:catalina.base}/logs/catalina.out
appender.RollingAppender.layout.type=PatternLayout
appender.RollingAppender.layout.pattern =%d %p [%t] [%c] [reqid:%X{XILC_req_id} app:%X{webappCtx} jsid:%X{jSessionId} aid:%X{XILC_aid} uid:%X{XILC_uid} org:%X{XILC_org} vorg:%X{XILC_vorg} sid:%X{XILC_sid} un:%X{XILC_un} sn:%X{XILC_sn}] %notEmpty{[jobName:%X{jobName}]}%notEmpty{[correlationId:%X{correlationId}]}%notEmpty{[jobHashCode:%X{jobHashCode}]} - %enc{%m}{CRLF}%n
appender.RollingAppender.policies.type=Policies

appender.RollingAppender.policies.size.type=SizeBasedTriggeringPolicy
appender.RollingAppender.policies.size.size=100MB
appender.RollingAppender.filePattern=${sys:catalina.base}/logs/catalina.out.%i{yyyy-MM-dd}
appender.RollingAppender.strategy.type=DefaultRolloverStrategy
appender.RollingAppender.strategy.max=1
appender.RollingAppender.strategy.action.type=Delete
appender.RollingAppender.strategy.action.basepath=${sys:catalina.base}/logs
appender.RollingAppender.strategy.action.maxdepth=1
appender.RollingAppender.strategy.action.PathConditions.type=IfFileName
appender.RollingAppender.strategy.action.PathConditions.glob=catalina.out.*
appender.RollingAppender.strategy.action.condition.type=IfAccumulatedFileCount
appender.RollingAppender.strategy.action.condition.exceeds=1

#Package loggers
logger.test.name=com.test
logger.test.level=DEBUG
logger.test.additivity=false
logger.test.appenderRefs=ar_rolling, MetricsAppenderRef
logger.test.appenderRef.ar_rolling.ref=CatalinaRollingAppender

Open Tracing Logger

logger.opentracing.name=com.test.cloud.service.opentracing
logger.opentracing.level=ERROR

#See hibernate HHH-9174. Statement#getWarnings can be an expensive call, should turn off by default by set log level to error

logger.hibernate.name=org.hibernate.engine.jdbc.spi.SqlExceptionHelper
logger.hibernate.level=ERROR
logger.hibernate.appenderRefs=ar_rolling, MetricsAppenderRef
logger.hibernate.ar_rolling.ref=CatalinaRollingAppender

Metrics Appender and logger settings

Metrics appender - this appender will be used to collect metrics related to log4j like error rate and Java exception metrics

This appender doesn't need any layout settings

appender.MetricsAppender.type=MetricsLog4j2ErrorAppender
appender.MetricsAppender.name=Metrics
appender.MetricsAppender.filter.threshold.type=ThresholdFilter
appender.MetricsAppender.filter.threshold.level=error

rootLogger.appenderRef.MetricsAppenderRef.ref=Metrics

logger.test.appenderRef.MetricsAppenderRef.ref=Metrics
logger.hibernate.appenderRef.MetricsAppenderRef.ref=Metrics

#Scheduler loggers
logger.schedulerOdata.name=com.test.service.scheduler.service.odata
logger.schedulerOdata.level=INFO

logger.flingo.name=com.test.cloud.flingo
logger.flingo.level=INFO

logger.schedulerDebugFilter.name=com.test.service.scheduler.filter.DebugFilter
logger.schedulerDebugFilter.level=INFO`

[jkwatson]

I don't see anything in here that would be configuring the logging bridge. I'm going to transfer this discussion over to the instrumentation repository where the folks there know how you would need to configure it to make it work.

[trask]

hi @AyushCarb1! can use the "logging" exporters, e.g.

OTEL_TRACES_EXPORTER=logging
OTEL_LOGS_EXPORTER=logging

and verify if your logs are being exported