Metrics from logs json parser issue

[nthomaslife360]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Hi,

I have am trying to parse a below event to Vector via s3 and trying to fetch metrics out of it.

#sample json event

I'm uploading this event to s3 and trying to get path and request_count.

It is getting ingested BUT while testing it by printing it out to internal console logs before creating metrics can see that each event message is getting parsed as single event, attached snippet below

Configuration

testing_te_file_remap:
        type: remap
        inputs:
          - route_s3_normalized.mobile_logs
        source: |-
           . = parse_json!(.message)

I want this to be parsed as single event like below

{"bucket":"life360-mobile-logs-dev","request_type":"mobile_logs","source_type":"aws_s3","message":"      

 {
    "path": "/",
    "request_count": 14,
    "average_duration": 1138.4285714285713,
    "total_data_transferred": 550994,
    "methods_distribution": [
      {
        "method": "POST",
        "count": 14
      }
    ]
  }
,"region":"us-east-1","object":"33.json"}

Version

"0.36.0-distroless-libc

Debug Output

No response

Example Data

Attached above

Additional Context

No response

References

No response

[jszwedko]

I converted this to a discussion. We are also discussing this in Discord: https://discord.com/channels/742820443487993987/1274053459213291611/1274058078144303236

[pront]

Your screenshot shows a JSON array spread across multiple lines. The source is splitting it into lines before parse_json sees it.

Use framing.method: bytes to read the whole object and let the JSON decoder produce one event per array element:

sources:
  mobile_logs:
    type: aws_s3
    region: us-east-1
    compression: auto
    sqs:
      queue_url: https://sqs.us-east-1.amazonaws.com/123456789012/mobile-logs
    framing:
      method: bytes
    decoding:
      codec: json

transforms:
  request_metrics:
    type: log_to_metric
    inputs: [mobile_logs]
    metrics:
      - type: counter
        field: request_count
        name: requests
        increment_by_value: true
        tags:
          path: "{{ path }}"

sinks:
  console:
    type: console
    inputs: [request_metrics]
    encoding:
      codec: json

Use your existing SQS notification queue. The decoded events already have .path and .request_count, so no parse_json remap is needed.

I tested this on Vector 0.60.0 with a gzipped copy of the two records in your screenshot and local S3/SQS mocks. It produced counter increments of 14 for /v2/ingest and 3 for /v5/ingest. This counter example assumes each record contains a new batch's request count.