Add support for basic metadata.yaml file

Author: jaydeluca

docs/contributing/writing-instrumentation.md

@@ -58,6 +58,7 @@ instrumentation ->
             build.gradle.kts
         testing
             build.gradle.kts
+        metadata.yaml
 ```
 
 The top level `settings.gradle.kts` file would contain the following (please add in alphabetical order):
@@ -68,6 +69,15 @@ include("instrumentation:yarpc-1.0:library")
 include("instrumentation:yarpc-1.0:testing")
 ```
 
+### Instrumentation metadata.yaml (Experimental)
+
+Each module can contain a `metadata.yaml` file that describes the instrumentation. This information
+is then used when generating the [instrumentation-list.yaml](../instrumentation-list.yaml) file.
+The schema for `metadata.yaml` is still in development and may change in the future. See the
+[instrumentation-docs readme](../../instrumentation-docs/readme.md) for more information and the
+latest schema.
+
+
 ### Instrumentation Submodules
 
 When writing instrumentation that requires submodules for different versions, the name of each
@@ -215,10 +225,10 @@ library instrumentation test, there will be code calling into the instrumentatio
 javaagent instrumentation test it will generally use the underlying library API as is and just rely
 on the javaagent to apply all the necessary bytecode changes automatically.
 
-You can use either JUnit 5 (recommended) or Spock to test the instrumentation. Start by creating an
-abstract class with an abstract method, for example `configure()`, that returns the instrumented
-object, such as a client, server, or the main class of the instrumented library. Then, depending on
-the chosen test library, go to the [JUnit](#junit) or [Spock](#spock) section.
+You can use JUnit 5 to test the instrumentation. Start by creating an abstract class with an
+abstract method, for example `configure()`, that returns the instrumented object, such as a client,
+server, or the main class of the instrumented library. See the [JUnit](#junit) section for more
+information.
 
 After writing some tests, return to the `library` package and make sure it has
 a `testImplementation` dependency on the `testing` submodule. Then, create a test class that extends
@@ -281,37 +291,6 @@ You can use the `@RegisterExtension` annotation to make sure that the instrument
 picked up by JUnit. Then, return the same extension instance in the `testing()` method
 implementation so that it's used in all test scenarios implemented in the abstract class.
 
-### Spock
-
-The `testing-common` module contains some utilities that facilitate writing Spock instrumentation
-tests, such as the `InstrumentationSpecification` base class and the `LibraryTestTrait`
-and `AgentTestTrait` traits.
-
-Consider the following abstract test class extending `InstrumentationSpecification`:
-
-```groovy
-abstract class AbstractYarpcTest extends InstrumentationSpecification {
-
-  abstract Yarpc configure(Yarpc yarpc);
-
-  def "test something"() {
-    // ...
-  }
-}
-```
-
-The `InstrumentationSpecification` class contains abstract methods that are implemented by one of
-our test traits in the actual test class. For example:
-
-```groovy
-class LibraryYarpcTest extends AbstractYarpcTest implements LibraryTestTrait {
-
-  @Override
-  Yarpc configure(Yarpc yarpc) {
-    // register interceptor/listener etc
-  }
-}
-```
 
 ## Writing Java agent instrumentation
 
@@ -352,9 +331,8 @@ without the user knowing about the instrumentation.
 
 Create a test that extends the base class you wrote earlier but does nothing in the `configure()`
 method. Unlike the library instrumentation, the javaagent instrumentation is supposed to work
-without any explicit user code modification. Depending on the testing framework, either use
-the `AgentInstrumentationExtension` or implement the `AgentTestingTrait`, and try running tests in
-this class. All tests should pass.
+without any explicit user code modification. Add an `AgentInstrumentationExtension` and try running
+tests in this class. All tests should pass.
 
 Note that all the tests inside the `javaagent` module are run using the `agent-for-testing`
 javaagent, with the instrumentation being loaded as an extension. This is done to perform the same

docs/instrumentation-list.yaml

@@ -204,10 +204,13 @@ cassandra:
   - name: cassandra-3.0
     srcPath: instrumentation/cassandra/cassandra-3.0
     target_versions:
-      javaagent: []
+      javaagent:
+      - com.datastax.cassandra:cassandra-driver-core:[3.0,4.0)
 clickhouse:
   instrumentations:
   - name: clickhouse-client-0.5
+    description: Instruments the V1 ClickHouseClient, providing database client spans
+      and metrics.
     srcPath: instrumentation/clickhouse-client-0.5
     target_versions:
       javaagent:

instrumentation-docs/readme.md

@@ -39,21 +39,36 @@ public class SpringWebInstrumentationModule extends InstrumentationModule
   }
 ```
 
-## Instrumentation meta-data
+## Instrumentation metadata
 
 * name
   * Identifier for instrumentation module, used to enable/disable
   * Configured in `InstrumentationModule` code for each module
-* versions
-  * List of supported versions by the module
-* type
-  * List of instrumentation types, options of either `library` or `javaagent`
+* srcPath
+  * Path to the source code of the instrumentation module
+* description
+  * Short description of what the instrumentation does
+* target_versions
+  * List of supported versions by the module, broken down by `library` or `javaagent` support
+
 
 ## Methodology
 
+### metadata.yaml file
+
+Within each instrumentation source directory, a `metadata.yaml` file can be created to provide
+additional information about the instrumentation module.
+
+As of now, the following fields are supported:
+
+```yaml
+description: "Description of what the instrumentation does."
+```
+
 ### Versions targeted
 
 We parse gradle files in order to determine the target versions.
 
 - Javaagent versions are determined by the `muzzle` plugin configurations
 - Library versions are determined by the library dependency versions
+  - when available, latestDepTestLibrary is used to determine the latest supported version

instrumentation-docs/src/main/java/io/opentelemetry/instrumentation/docs/InstrumentationAnalyzer.java

@@ -9,6 +9,7 @@
 
 import io.opentelemetry.instrumentation.docs.utils.FileManager;
 import io.opentelemetry.instrumentation.docs.utils.InstrumentationPath;
+import io.opentelemetry.instrumentation.docs.utils.YamlHelper;
 import java.util.ArrayList;
 import java.util.HashMap;
 import java.util.HashSet;
@@ -53,8 +54,9 @@ public static List<InstrumentationEntity> convertToEntities(List<Instrumentation
   }
 
   /**
-   * Analyzes the given root directory to find all instrumentation paths and then analyze them. -
-   * Extracts version information from each instrumentation's build.gradle file.
+   * Analyzes the given root directory to find all instrumentation paths and then analyze them.
+   * Extracts version information from each instrumentation's build.gradle file. Extracts
+   * information from metadata.yaml files.
    *
    * @return a list of InstrumentationEntity objects with target versions
    */
@@ -65,6 +67,11 @@ List<InstrumentationEntity> analyze() {
     for (InstrumentationEntity entity : entities) {
       List<String> gradleFiles = fileSearch.findBuildGradleFiles(entity.getSrcPath());
       analyzeVersions(gradleFiles, entity);
+
+      String metadataFile = fileSearch.getMetaDataFile(entity.getSrcPath());
+      if (metadataFile != null) {
+        entity.setMetadata(YamlHelper.metaDataParser(metadataFile));
+      }
     }
     return entities;
   }

instrumentation-docs/src/main/java/io/opentelemetry/instrumentation/docs/InstrumentationEntity.java

@@ -13,6 +13,8 @@ public class InstrumentationEntity {
   private final String instrumentationName;
   private final String namespace;
   private final String group;
+
+  private InstrumentationMetaData metadata;
   private Map<InstrumentationType, Set<String>> targetVersions;
 
   public InstrumentationEntity(
@@ -28,12 +30,22 @@ public InstrumentationEntity(
       String instrumentationName,
       String namespace,
       String group,
-      Map<InstrumentationType, Set<String>> targetVersions) {
+      Map<InstrumentationType, Set<String>> targetVersions,
+      InstrumentationMetaData metadata) {
     this.srcPath = srcPath;
     this.instrumentationName = instrumentationName;
     this.namespace = namespace;
     this.group = group;
     this.targetVersions = targetVersions;
+    this.metadata = metadata;
+  }
+
+  public void setMetadata(InstrumentationMetaData metadata) {
+    this.metadata = metadata;
+  }
+
+  public InstrumentationMetaData getMetadata() {
+    return metadata;
   }
 
   public String getSrcPath() {

instrumentation-docs/src/main/java/io/opentelemetry/instrumentation/docs/InstrumentationMetaData.java

@@ -0,0 +1,25 @@
+/*
+ * Copyright The OpenTelemetry Authors
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+package io.opentelemetry.instrumentation.docs;
+
+public class InstrumentationMetaData {
+
+  public InstrumentationMetaData() {}
+
+  public InstrumentationMetaData(String description) {
+    this.description = description;
+  }
+
+  private String description;
+
+  public String getDescription() {
+    return description;
+  }
+
+  public void setDescription(String description) {
+    this.description = description;
+  }
+}

instrumentation-docs/src/main/java/io/opentelemetry/instrumentation/docs/utils/FileManager.java

@@ -102,6 +102,14 @@ public List<String> findBuildGradleFiles(String instrumentationDirectory) {
     }
   }
 
+  public String getMetaDataFile(String instrumentationDirectory) {
+    String metadataFile = instrumentationDirectory + "/metadata.yaml";
+    if (Files.exists(Paths.get(metadataFile))) {
+      return readFileToString(metadataFile);
+    }
+    return null;
+  }
+
   public String readFileToString(String filePath) {
     try {
       return Files.readString(Paths.get(filePath));

instrumentation-docs/src/main/java/io/opentelemetry/instrumentation/docs/utils/YamlHelper.java

@@ -6,8 +6,10 @@
 package io.opentelemetry.instrumentation.docs.utils;
 
 import io.opentelemetry.instrumentation.docs.InstrumentationEntity;
+import io.opentelemetry.instrumentation.docs.InstrumentationMetaData;
 import java.io.BufferedWriter;
 import java.util.ArrayList;
+import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.TreeMap;
@@ -29,21 +31,25 @@ public static void printInstrumentationList(
     Map<String, Object> output = new TreeMap<>();
     groupedByGroup.forEach(
         (group, entities) -> {
-          Map<String, Object> groupMap = new TreeMap<>();
+          Map<String, Object> groupMap = new LinkedHashMap<>();
           List<Map<String, Object>> instrumentations = new ArrayList<>();
           for (InstrumentationEntity entity : entities) {
-            Map<String, Object> entityMap = new TreeMap<>();
+            Map<String, Object> entityMap = new LinkedHashMap<>();
             entityMap.put("name", entity.getInstrumentationName());
+
+            if (entity.getMetadata() != null && entity.getMetadata().getDescription() != null) {
+              entityMap.put("description", entity.getMetadata().getDescription());
+            }
+
             entityMap.put("srcPath", entity.getSrcPath());
 
             Map<String, Object> targetVersions = new TreeMap<>();
             if (entity.getTargetVersions() != null && !entity.getTargetVersions().isEmpty()) {
               entity
                   .getTargetVersions()
                   .forEach(
-                      (type, versions) -> {
-                        targetVersions.put(type.toString(), new ArrayList<>(versions));
-                      });
+                      (type, versions) ->
+                          targetVersions.put(type.toString(), new ArrayList<>(versions)));
             }
             entityMap.put("target_versions", targetVersions);
             instrumentations.add(entityMap);
@@ -60,5 +66,9 @@ public static void printInstrumentationList(
     yaml.dump(output, writer);
   }
 
+  public static InstrumentationMetaData metaDataParser(String input) {
+    return new Yaml().loadAs(input, InstrumentationMetaData.class);
+  }
+
   private YamlHelper() {}
 }

instrumentation-docs/src/test/java/io/opentelemetry/instrumentation/docs/utils/YamlHelperTest.java

@@ -8,6 +8,7 @@
 import static org.assertj.core.api.Assertions.assertThat;
 
 import io.opentelemetry.instrumentation.docs.InstrumentationEntity;
+import io.opentelemetry.instrumentation.docs.InstrumentationMetaData;
 import io.opentelemetry.instrumentation.docs.InstrumentationType;
 import java.io.BufferedWriter;
 import java.io.StringWriter;
@@ -21,20 +22,24 @@
 
 class YamlHelperTest {
   @Test
-  public void testPrintInstrumentationList() throws Exception {
+  void testPrintInstrumentationList() throws Exception {
     List<InstrumentationEntity> entities = new ArrayList<>();
     Map<InstrumentationType, Set<String>> targetVersions1 = new HashMap<>();
     targetVersions1.put(
         InstrumentationType.JAVAAGENT,
         new HashSet<>(List.of("org.springframework:spring-web:[6.0.0,)")));
 
+    InstrumentationMetaData metadata1 =
+        new InstrumentationMetaData("Spring Web 6.0 instrumentation");
+
     entities.add(
         new InstrumentationEntity(
             "instrumentation/spring/spring-web/spring-web-6.0",
             "spring-web-6.0",
             "spring",
             "spring",
-            targetVersions1));
+            targetVersions1,
+            metadata1));
 
     Map<InstrumentationType, Set<String>> targetVersions2 = new HashMap<>();
     targetVersions2.put(
@@ -46,7 +51,8 @@ public void testPrintInstrumentationList() throws Exception {
             "struts-2.3",
             "struts",
             "struts",
-            targetVersions2));
+            targetVersions2,
+            null));
 
     StringWriter stringWriter = new StringWriter();
     BufferedWriter writer = new BufferedWriter(stringWriter);
@@ -58,6 +64,7 @@ public void testPrintInstrumentationList() throws Exception {
         "spring:\n"
             + "  instrumentations:\n"
             + "  - name: spring-web-6.0\n"
+            + "    description: Spring Web 6.0 instrumentation\n"
             + "    srcPath: instrumentation/spring/spring-web/spring-web-6.0\n"
             + "    target_versions:\n"
             + "      javaagent:\n"

instrumentation/cassandra/cassandra-3.0/javaagent/build.gradle.kts

@@ -2,14 +2,11 @@ plugins {
   id("otel.javaagent-instrumentation")
 }
 
-val cassandraDriverTestVersions = "[3.0,4.0)"
-
 muzzle {
-
   pass {
     group.set("com.datastax.cassandra")
     module.set("cassandra-driver-core")
-    versions.set(cassandraDriverTestVersions)
+    versions.set("[3.0,4.0)")
     assertInverse.set(true)
   }
 
@@ -19,7 +16,7 @@ muzzle {
     name.set("Newest versions of Guava")
     group.set("com.datastax.cassandra")
     module.set("cassandra-driver-core")
-    versions.set(cassandraDriverTestVersions)
+    versions.set("[3.0,4.0)")
     // While com.datastax.cassandra uses old versions of Guava, users may depends themselves on newer versions of Guava
     extraDependency("com.google.guava:guava:27.0-jre")
   }

instrumentation/clickhouse-client-0.5/metadata.yaml

@@ -0,0 +1 @@
+description: Instruments the V1 ClickHouseClient, providing database client spans and metrics.