Inject and extract custom field (key,value) to/from the Propagators

[itsmaneesh]

I have use case to set a custom field to the Propagators so that this field (into a baggage) would get propagated across services that are already auto-instrumented.

For e.g.: If the request flows through services S1 -> S2 -> S3 -> S4, I would like to set a custom field at the S1 service and the following services can use this field to perform a custom logic such log level update, custom validation... etc.

Can you give me the references (with example usage) to implement this feature? Would it be possible to utilize the Baggage API or by some other means?

[trask]

yes, the Baggage API (and the baggage propagator) should work well for this use case

[itsmaneesh]

@trask Can you give me reference to use this Baggage API (usage examples)?

[trask]

sure, check out https://github.com/open-telemetry/opentelemetry-java/blob/main/api/all/src/main/java/io/opentelemetry/api/baggage/Baggage.java

[itsmaneesh]

I was trying to use the W3CBaggagePropagator to inject and extract custom field in/out of Baggage shared between two service. But, looks like Context Propagation is not happening in my use case mentioned below:
For e.g.:
I have two simple REST services that uses auto-instrumentation and a simple library that can be shared as dependency between two services.
util-lib has the logic to inject and extract a custom field in/out of Baggage.

util-lib has the below class that would be used by both services as gradle dependency.

package com.myorg.alpha.util.logging.builder;

import io.opentelemetry.api.baggage.Baggage;
import io.opentelemetry.api.baggage.propagation.W3CBaggagePropagator;
import io.opentelemetry.context.Context;
import io.opentelemetry.context.propagation.TextMapGetter;
import org.apache.logging.log4j.LogManager;

import javax.annotation.Nullable;
import java.util.HashMap;
import java.util.Map;

public final class BaggageUtil {
    private static final W3CBaggagePropagator baggagePropagator = W3CBaggagePropagator.getInstance();
    private static final Map<String, String> carrier = new HashMap<>();

    
    public static void injectCustomLogLevelIntoBaggage(String customLogLevel) {
        Baggage baggage = Baggage.builder()
                    .put(AlphaConfigurationFactory.ALPHA_LOGGER_ROUTE_KEY, customLogLevel).build();
        baggagePropagator.inject(Context.current().with(baggage), carrier, Map::put);

    }

    public static String extractCustomLogLevelFromBaggage() {
        Context extractedContext = baggagePropagator.extract(Context.current(), carrier, new MapTextMapGetter());
        Baggage extractedBaggage = Baggage.fromContext(extractedContext);
        return extractedBaggage.getEntryValue(AlphaConfigurationFactory.ALPHA_LOGGER_ROUTE_KEY);
    }

    private static class MapTextMapGetter implements TextMapGetter<Map<String, String>> {
        @Override
        public Iterable<String> keys(Map<String, String> carrier) {
            return carrier.keySet();
        }

        @Nullable
        @Override
        public String get(Map<String, String> carrier, String key) {
            return carrier.get(key);
        }
    }
}

user-service injects a custom field into the Baggage using util-lib and then invokes department-service using RestTemplate.

@GetMapping("{userId}")
    public ResponseEntity<String> getDepartmentsForUser(@PathVariable("userId") String userId) {
        BaggageUtil.injectCustomLogLevelIntoBaggage("CUSTOM_LOG_LEVEL_FROM_USER_SERVICE");
        String responseEntityFromDepartmentService = restTemplate.getForObject("http://localhost:8084/api/departments/"+userId, String.class);
        return ResponseEntity.ok(responseEntityFromDepartmentService);
    }

dept-service receives the call from user-service and the call is handled as shown below:

@GetMapping("{userId}")
    public ResponseEntity<String> getDepartmentsForUser(@PathVariable("userId") int userId) {
        String levelFromBaggage = BaggageUtil.extractCustomLogLevelFromBaggage();
        System.out.println("************inside department-service, levelFromBaggage: "+ levelFromBaggage);
        return ResponseEntity.ok(getResponseString(userId));
    }

But, I could see the dept-service DID NOT receive the expected Context from user-service as the log statement shows the below content in the Context object (invoked from dept-service):

2023-11-30T17:39:13.818-05:00  INFO 67188 --- [nio-8084-exec-1] c.c.a.util.logging.builder.BaggageUtil   : extracted context: AgentContextWrapper{agentContext={}, applicationContext={}}

Can you please guide me in resolving this issue here so that context would get propagated properly.
@laurit @trask

[laurit]

@GetMapping("{userId}")
public ResponseEntity<String> getDepartmentsForUser(@PathVariable("userId") String userId) {
  Baggage baggage = Baggage.builder().put("CUSTOM_LOG_LEVEL_FROM_USER_SERVICE", "test").build();
  try (Scope ignored = baggage.makeCurrent()) {
    String responseEntityFromDepartmentService = restTemplate.getForObject("http://localhost:8084/api/departments/"+userId, String.class);
    return ResponseEntity.ok(responseEntityFromDepartmentService);
  }
}

@GetMapping("{userId}")
public ResponseEntity<String> getDepartmentsForUser(@PathVariable("userId") int userId) {
  String levelFromBaggage = Baggage.current().getEntryValue("CUSTOM_LOG_LEVEL_FROM_USER_SERVICE");
  System.out.println("************inside department-service, levelFromBaggage: "+ levelFromBaggage);
  return ResponseEntity.ok(getResponseString(userId));
}

[itsmaneesh]

@laurit Thank you for the input and above approach is working. If you could provide the input to the below as well, that will be great.

• Can this same approach be used for gRPC, Kafka Streams ...etc frameworks as well? Or is there any variation to the above approach?
• I could see that my first service can add multiple <key , value> pairs say <key1FromService1,val1FromService1> , <key2FromService1, value2FromService1> to the Baggage builder and build a Baggage object then second service would be able to access these multiple <key , value> pairs from Baggage.current().getEntryValue().

But, If my second service need to set additional <key , value> pair say <key1FromService2 value1FromService2> to this Baggage so that the downstream services (say third service) can access all the three <key , value> pairs. How can we achieve this?

For e.g.:

First Service (user-service):

@GetMapping("{userId}")
public ResponseEntity<String> getDepartmentsForUser(@PathVariable("userId") String userId) {
  Baggage baggage = Baggage.builder().put("key1FromService1", "val1FromService1")
                                     .put("key2FromService1", "val2FromService1")
                                     .build();
  try (Scope ignored = baggage.makeCurrent()) {
    String responseEntityFromDepartmentService = restTemplate.getForObject("http://localhost:8084/api/departments/"+userId, String.class);
    return ResponseEntity.ok(responseEntityFromDepartmentService);
  }
}

Second Service (dept-service):

@GetMapping("{userId}")
public ResponseEntity<String> getDepartmentsForUser(@PathVariable("userId") int userId) {
  Baggage baggage = Baggage.builder().put("key1FromService2", "val1FromService2").build();
        try(Scope ignored = baggage.makeCurrent()) {
            String key1FromService1= Baggage.current().getEntryValue("key1FromService1");
            System.out.println("************inside department-service, key1FromService1: " + key1FromService1);
            String key2FromService= Baggage.current().getEntryValue("key2FromService1");
            System.out.println("************inside department-service, key2FromService1: " + key2FromService1);
            String responseEntityFromLocationService = restTemplate.getForObject("http://localhost:8085/api/locations/" + DepartmentServiceUtil.fetchDepartmentsForUser(userId).toString(), String.class);
            return ResponseEntity.ok(getResponseString(userId, responseEntityFromLocationService));
        }
}

Third Service (location-service):

@GetMapping("{deptname}")
    public ResponseEntity<String> getLocationsForDepartment(@PathVariable("deptname") String deptname) {
        String key1FromService1= Baggage.current().getEntryValue("key1FromService1");
        System.out.println("************inside location-service, key1FromService1: " + key1FromService1);
        String key2FromService1= Baggage.current().getEntryValue("key2FromService1");
        System.out.println("************inside location-service, key2FromService1: " + key2FromService1);
        String key1FromService2= Baggage.current().getEntryValue("key1FromService2");
        System.out.println("************inside location-service, key1FromService2: " + key1FromService2);
        return ResponseEntity.ok(getResponseString(deptname));
    }

• Documentation reference link to the above approach. (So that I could provide pointers in our internal org documentation to Otel documentation page).