Elasticsearch 9.0.1: media_type_header_exception

[terebentina]

Hi folks,

I'm trying my first pipeline with redpanda connect with a very simple scenario: pick from kafka, save to elasticsearch.
However, I get an error message I have no idea how to solve: Failed to send message to elasticsearch_v8: sending bulk request: status: 400, failed: [media_type_header_exception], reason: Invalid media-type value on headers [Accept, Content-Type].

My elasticsearch server is 9.0.1 and I am using the elasticsearch_v8 output connector. I'm thinking it's because of the version mismatch but not sure since I can't see the actual headers.
Have you tested this against 9.x servers or is there any plan on releasing an elasticsearch_v9 connector?

My output config is really simple:

output:
  label: ""
  elasticsearch_v8:
    urls:
      - ${ES_URL}
    index: stream-v1-${! meta("org") }
    action: "create"
    id: ${!counter()}-${!timestamp_unix()}
    max_in_flight: 100
    tls:
      enabled: true
      skip_cert_verify: true

[mihaitodor]

Hey @terebentina, unfortunately it looks like there are a few data validation issues in that component that we should address. The only valid values for action right now are index, update and delete (see here) while the deprecated (and now removed) elasticsearch output did support create and upsert as well (see here). I'll convert this into an issue since we should address these gaps or at the very least add some validation.

LE: Follow-up issue: #3443