Label selectors break reconciliation - attempts to CREATE existing topics

[Yoni-Weisberg]

Describe the bug

When using label selectors (e.g., --selector 'label: my-label IN (...)'), Jikkou fails to recognize existing Kafka topics and attempts to CREATE them instead of performing UPDATE/NONE operations. This results in TopicExistsException errors. Name-based and kind-based selectors work correctly.

The root cause is that label selectors are applied to both expected resources (from templates, which have labels) AND actual resources (from Kafka, which don't have custom labels). This causes actual topics to be filtered out during reconciliation, breaking the name-based matching logic.

To Reproduce

1. Create a topic in Kafka:

kafka-topics.sh --create --topic my-topic --partitions 1 --replication-factor 1

2. Create a Jikkou template defining the same topic with a custom label:

apiVersion: "kafka.jikkou.io/v1"
kind: "KafkaTopic"
metadata:
  name: my-topic
  labels:
    my-label: "my-service"
spec:
  partitions: 1
  replicas: 1

3. Run Jikkou with a label selector:

jikkou apply --files templates/ --selector 'label: my-label IN (my-service)'

Expected behavior

Jikkou should recognize that my-topic already exists in Kafka and either:

• Perform NONE operation (no changes needed), or
• Perform UPDATE operation (if configuration differs)

The label selector should only filter which resources from templates are managed, not affect the matching logic against existing Kafka resources.

Actual behavior

Jikkou attempts to CREATE the topic, resulting in:

TopicExistsException: Topic 'my-topic' already exists.

The reconciliation fails because:

1. Expected topics (from templates) have the custom label and pass the selector
2. Actual topics (from Kafka) don't have custom labels and are filtered OUT by the selector
3. Name-based matching fails because actual list is empty
4. Jikkou thinks the topic doesn't exist and tries to CREATE it

Root Cause Analysis

In AdminClientKafkaTopicController.plan() (lines 121-144), the selector is applied to both lists:

// Expected resources - selector applied
List<V1KafkaTopic> expectedKafkaTopics = resources.stream()
    .filter(context.selector()::apply)  // Filters by label
    .toList();

// Actual resources - selector also applied
List<V1KafkaTopic> actualKafkaTopics = collector.listAll()
    .stream()
    .filter(context.selector()::apply)  // Also filters by label!
    .toList();

In KafkaTopicService.newTopicResources() (lines 94-101), topics fetched from Kafka only get:

• metadata.name from the topic name
• jikkou.io/kafka.topic.id label

Custom labels don't exist because Kafka doesn't store them. They only exist in template definitions.

Workaround

Use selectors that work on properties that exist in both templates and Kafka:

• --selector 'kind IN (KafkaTopic)' ✓
• --selector 'metadata.name IN (topic1, topic2)' ✓
• Avoid label-based selectors for filtering ✗

Screenshots/Configs

# Template with custom label
apiVersion: "kafka.jikkou.io/v1"
kind: "KafkaTopic"
metadata:
  name: test-topic
  labels:
    my-label: "my-value"  # This label only exists in template
spec:
  partitions: 1
  replicas: 1

Runtime environment

• OS: Ubuntu 22.04 / Windows WSL2
• Jikkou: 0.37.0
• Kafka Cluster Version: 3.9.1

Additional context

This issue affects any use case where custom labels are used to organize or categorize topics in templates for selective management. It makes label-based resource organization impractical for Jikkou workflows.

Suggested Fix

Apply selectors only to expected resources (from templates), not to actual resources (from Kafka):

List<V1KafkaTopic> expectedKafkaTopics = resources.stream()
    .filter(context.selector()::apply)  // Apply selector
    .toList();

List<V1KafkaTopic> actualKafkaTopics = collector.listAll()
    .stream()
    // .filter(context.selector()::apply)  // DON'T apply selector here
    .toList();

[fhussonnois]

Hey @Yoni-Weisberg thank you for opening this issue and all the details. A fixed was released. Basically, labels are now propagated from expected resources to matching actual resources before applying the selector, so both sides are filtered consistently. This is a "best-effort" mechanism as Apache Kafka does not have any concept of tags or labels for topics :(

[Yoni-Weisberg]

@fhussonnois I'm still hitting this issue with connectors, so the bug seems to be broader than we thought. It likely applies to schemas too—might need a more global fix since the current one appears partial