How to read csv response with a Custom Connector using low-code CDK

[Juan Carlos Carvajal B (jucajuca)]

Hello,
I want to create a custom connector for a CSV file which is fetched through a HTTP request with API-Key Authentication. I managed to get the Authentication working, but I do not understand how can I tell airbyte that it should expect a CSV file (and not a json file).

Basically I just need the same file-connector but with an API-Key mechanism

Can you give me some hints on how to do this? Much appreciated.

version: 0.44.4
type: DeclarativeSource
check:
  type: CheckStream
  stream_names:
    - csv
streams:
  - type: DeclarativeStream
    name: csv
    primary_key: []
    schema_loader:
      type: InlineSchemaLoader
      schema: {}
    retriever:
      type: SimpleRetriever
      requester:
        type: HttpRequester
        url_base: https://app.domain.tld/api/
        path: /docs/2ZSyHmrYeDHpcQpVtacGBn/download/csv
        http_method: GET
        request_parameters:
          tableId: Cities
        request_headers: {}
        authenticator:
          type: BearerAuthenticator
          api_token: '{{ config[''api_key''] }}'
        request_body_json: {}
      record_selector:
        type: RecordSelector
        extractor:
          type: DpathExtractor
          field_path: []
      paginator:
        type: NoPagination
  - type: DeclarativeStream
    name: sqlite
    primary_key: []
    schema_loader:
      type: InlineSchemaLoader
      schema:
        $schema: http://json-schema.org/draft-07/schema#
        additionalProperties: true
        properties: {}
        type: object
    retriever:
      type: SimpleRetriever
      requester:
        type: HttpRequester
        url_base: https://app.domain.tld/api/
        path: /docs/2ZSyHmrYeDHpcQpVtacGBn/download
        http_method: GET
        request_parameters: {}
        request_headers: {}
        authenticator:
          type: BearerAuthenticator
          api_token: '{{ config[''api_key''] }}'
        request_body_json: {}
      record_selector:
        type: RecordSelector
        extractor:
          type: DpathExtractor
          field_path: []
      paginator:
        type: NoPagination
spec:
  connection_specification:
    $schema: http://json-schema.org/draft-07/schema#
    type: object
    required:
      - api_key
    properties:
      api_key:
        type: string
        title: API Key
        airbyte_secret: true
        order: 0
    additionalProperties: true
  documentation_url: https://example.org
  type: Spec
metadata:
  autoImportSchema:
    csv: true
    sqlite: true

[M. Marx (marcosmarxm)]

Airbyte's low-code framework is primarily designed to handle JSON responses from REST APIs. However, if the API response is a CSV file, you can still handle it by writing a custom component to override the extractor.
In the YAML configuration for your custom connector, you would need to replace the DpathExtractor, which is designed to extract JSON records, with your custom CSV extractor. Here's an example of what that might look like:

yaml
record_selector:
  type: RecordSelector
  extractor:
    type: YourCustomCSVExtractor
    field_path: []

Your custom CSV extractor should convert the CSV into a list of records (dictionaries in Python). You can refer to the DpathExtractor's implementation for an idea of how to structure your custom extractor.
For more information on writing custom components and how to use them, refer to the Airbyte documentation.
Please note that this approach requires some coding and is not entirely low-code. If you need a completely no-code solution, you might need to use a different tool or service that can convert the CSV response to JSON before it's processed by Airbyte.