Allow disabling builtin tags when reporting custom metrics

[eyalroth]

When reporting custom metrics, some tags/dimensions are reported by default with the metric; for instance, the lambda name (functionname).

This means that different lambda functions cannot populate the same metric instance (time series).

This is not such an issue for basic metrics -- max, min, count, etc -- but is a major issue for percentile metrics, since those timeseries cannot be aggregated over (despite datadog's UI allowing this, which is very misleading, but that's another topic).

For instance, if 2 of my lambdas make API calls to some third party service, and I want to aggregate and measure the overall percentile of these calls across all of my lambdas, I simply cannot do it with datadog.

Please add an option to remove some/all builtin tags when reporting custom metrics from Lambda. That is, assuming the Datadog backend supports this.

[shreyamalpani]

Hi Eyal,

You should be able to aggregate percentile metrics across Lambda functions - the built-in functionname tag can be used as a filter but does not prevent aggregation on the backend.

As long there is not a filter or group by on functionname in your queries or dashboards, the custom metrics percentiles should be correctly aggregated across functions.

Let me know if you have any other questions!

[eyalroth]

You should be able to aggregate percentile metrics across Lambda functions

@shreyamalpani Thanks for the reply!

I just tested it and it does work as expected, and as documented in this blog post about DDSketch.

My test code is below, but in short, what it does:

• every X seconds generate Y datapoints between 0 and 10K (representing duration in millis), with some pseudo algorithm that tries to mimic request latency distribution
• Associate every datapoint with one of 3 groups according to probability (I used 50%, 30%, 20%)
• Report every datapoint individually using both AWS EMF and DD
• For EMF the report is twofold -- once with the group name in the dimension, one without (total aggregate)

I made sure to run the code sequentially (i.e only one lambda container), and to enable "percentile metrics" for the metric in DD.

The exact values of the percentiles -- both for the total aggregation and per-group -- were different between AWS and DD. This is expected since both use some approximation (I imagine), and the algorithm might vary. However, the values were in the same ballpark and followed the same trend, so that's good.

I think that my confusion stemmed from the fact that imported metrics, such as AWS builtin metrics (e.g lambda duration), do not aggregate properly. I'm guessing it's because the "sketches" are not available on the source metrics (since they weren't originally generated by DD), so DD cannot merge them correctly.

Do you know of any way to enable this for AWS builtin metrics?

---

The test code:

import { getLogger } from 'log4js';
import * as awsMetrics from 'aws-embedded-metrics';
import { sendDistributionMetric } from 'datadog-lambda-js';

type Input = {
  totalTimeMillis: number;
  datapointsPerInterval: number;
  intervalMillis: number;
  groups: Record<string, number>;
};
type Metrics = Record<string, number[]>;

const METRIC_NAME_DD = 'eyalroth.metrics-test.1';
const METRIC_NAME_AWS = METRIC_NAME_DD.replace('.', '_');

const logger = getLogger('metrics-test');

awsMetrics.Configuration.namespace = 'eyalroth-metrics-test';
const rootMetricsLogger = awsMetrics.createMetricsLogger();

export const handler = async (event: Input) => {
  await processMetrics(event);
};

async function processMetrics(input: Input): Promise<void> {
  const { totalTimeMillis, datapointsPerInterval, intervalMillis, groups } = input;

  let keepGoing = true;

  setTimeout(() => {
    keepGoing = false;
  }, totalTimeMillis);

  while (keepGoing) {
    const datapoints = generateDatapoints(datapointsPerInterval);
    const metrics = associateWithGroups(datapoints, groups);
    await report(metrics);

    await new Promise((resolve) => setTimeout(resolve, intervalMillis));
  }
}

function generateDatapoints(count: number): number[] {
  const datapoints: number[] = [];

  for (let i = 0; i < count; i++) {
    const baseLatency = Math.max(0, Math.random() * 200);
    const spike = Math.random() > 0.8 ? Math.random() * Math.random() * 9800 : 0;

    const latency = Math.min(10000, Math.floor(baseLatency + spike));
    datapoints.push(latency);
  }

  return datapoints;
}

async function report(metricsPerGroup: Metrics): Promise<void> {
  logger.info('Reporting metrics');

  logger.info('Reporting EMF');
  for (const [groupName, datapoints] of Object.entries(metricsPerGroup)) {
    for (const point of datapoints) {
      await rootMetricsLogger.new().putDimensions({ groupName }).putMetric(METRIC_NAME_AWS, point, awsMetrics.Unit.Milliseconds).flush();
      await rootMetricsLogger.new().putMetric(METRIC_NAME_AWS, point, awsMetrics.Unit.Milliseconds).flush();
    }
  }

  logger.info('Reporting Datadog');
  for (const [groupName, datapoints] of Object.entries(metricsPerGroup)) {
    for (const point of datapoints) {
      sendDistributionMetric(METRIC_NAME_DD, point, `groupName:${groupName}`);
    }
  }

  const allMetrics = Object.values(metricsPerGroup).flatMap((m) => m);
  allMetrics.sort((a, b) => a - b);
  const metrics = {
    ...metricsPerGroup,
    allMetrics,
  };
  logger.info('Metrics reported', metrics);
}

function associateWithGroups(datapoints: number[], groups: Record<string, number>): Record<string, number[]> {
  const result: Record<string, number[]> = {};
  const groupNames = Object.keys(groups);

  groupNames.forEach((name) => {
    result[name] = [];
  });

  datapoints.forEach((datapoint) => {
    const random = Math.random();
    let cumulativeProbability = 0;

    for (const groupName of groupNames) {
      cumulativeProbability += groups[groupName];

      if (random <= cumulativeProbability) {
        result[groupName].push(datapoint);
        break;
      }
    }
  });

  return result;
}

[shreyamalpani]

Thanks for the additional context! That's correct, the builtin AWS metrics are sent as Gauge / Count metrics and do not support percentile aggregation. The enhanced metrics generated by the Datadog Lambda Extension (i.e. aws.lambda.enhanced.duration) support percentile aggregation across functions as they are sent as Distribution metrics, which are ingested as sketches.

[eyalroth]

@shreyamalpani yeah the problem with the extension metrics is that they don't account for start time of other extensions (during cold start).

Also, a bit off topic, the most important duration metric for us is API gateway requests latency, which again the extension cannot account for. Do you know if there are other solutions for that?