Confluent Platform (cp-all-in-one)

• Source: https://github.com/confluentinc/cp-all-in-one

Setup

• Copy .env.example to .env & set env below
  • AWS Access Key must have access AmazonS3FullAccess, AWSGlueConsoleFullAccess

AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=

• Remove interceptor class
  • Do not set Confluent interceptor classes—these are not in the community image.
  • For ksqldb-server, set these in docker-compose.yml:

KSQL_PRODUCER_INTERCEPTOR_CLASSES: ""
KSQL_CONSUMER_INTERCEPTOR_CLASSES: ""

• Comment out to use default partition 12 due to Confluent-image bugs

# CONTROL_CENTER_INTERNAL_TOPICS_PARTITIONS: 1
# CONTROL_CENTER_MONITORING_INTERCEPTOR_TOPIC_PARTITIONS: 1

• Start all services with

docker-compose up -d

# ----- OR ----- 

docker compose up -d broker
docker compose up -d schema-registry
docker compose up -d ksqldb-server
docker compose up -d rest-proxy
docker compose up -d ksqldb-cli
docker compose up -d prometheus
docker compose up -d alertmanager
docker compose up -d control-center
docker compose up -d flink-jobmanager
docker compose up -d flink-taskmanager
docker compose up -d flink-sql-client

Access to Flink-SQL-Client

• Run command below

docker exec -it flink-sql-client bash
sql-client.sh

Persistent Catalog Store

• This is to set Catelog to be persistent instead of session-based
• Add config below to /opt/flink/conf/flink-conf.yaml

table.catalog-store.kind: file
table.catalog-store.file.path: /app/catalog_store