chore(qdrant): various fixes

README.md

@@ -34,7 +34,7 @@ KubeBlocks add-ons.
 | polardbx | polardbx-cdc-2.3.0<br>polardbx-cn-2.3.0<br>polardbx-dn-2.3.0<br>polardbx-gms-2.3.0 | PolarDB-X is a cloud native distributed SQL Database designed for high concurrency, massive storage, complex querying scenarios. | ApeCloud Vettal Wu |
 | postgresql | postgresql-12.14.0<br>postgresql-12.14.1<br>postgresql-12.15.0<br>postgresql-12.22.0<br>postgresql-13.23.0<br>postgresql-14.18.0<br>postgresql-14.23.0<br>postgresql-14.7.2<br>postgresql-14.8.0<br>postgresql-15.13.0<br>postgresql-15.18.0<br>postgresql-15.7.0<br>postgresql-16.14.0<br>postgresql-16.4.0<br>postgresql-16.9.0<br>postgresql-17.10.0<br>postgresql-17.5.0<br>postgresql-18.1.0<br>postgresql-18.4.0 | A PostgreSQL (with Patroni HA) cluster definition Helm chart for Kubernetes | ldming dingshun-cmss |
 | pulsar | pulsar-bookies-recovery-2.11.2<br>pulsar-bookies-recovery-3.0.2<br>pulsar-bookies-recovery-4.0.6<br>pulsar-bookkeeper-2.11.2<br>pulsar-bookkeeper-3.0.2<br>pulsar-bookkeeper-4.0.6<br>pulsar-broker-2.11.2<br>pulsar-broker-3.0.2<br>pulsar-broker-4.0.6<br>pulsar-proxy-2.11.2<br>pulsar-proxy-3.0.2<br>pulsar-proxy-4.0.6<br>pulsar-zookeeper-2.11.2<br>pulsar-zookeeper-3.0.2 | Apache Pulsar is an open-source, distributed messaging and streaming platform built for the cloud. | cjc7373 caiq1nyu |
-| qdrant | qdrant-1.10.0<br>qdrant-1.13.4<br>qdrant-1.15.4<br>qdrant-1.15.5<br>qdrant-1.16.3<br>qdrant-1.17.1<br>qdrant-1.18.2<br>qdrant-1.5.0<br>qdrant-1.7.3<br>qdrant-1.8.1<br>qdrant-1.8.4 | High-performance, massive-scale Vector Database for the next generation of AI. | cjc7373 |
+| qdrant | qdrant-1.10.0<br>qdrant-1.13.4<br>qdrant-1.15.4<br>qdrant-1.15.5<br>qdrant-1.16.3<br>qdrant-1.17.1<br>qdrant-1.18.2<br>qdrant-1.19.0<br>qdrant-1.5.0<br>qdrant-1.7.3<br>qdrant-1.8.1<br>qdrant-1.8.4 | High-performance, massive-scale Vector Database for the next generation of AI. | cjc7373 |
 | rabbitmq | rabbitmq-3.10.25<br>rabbitmq-3.11.28<br>rabbitmq-3.12.14<br>rabbitmq-3.13.7<br>rabbitmq-3.8.34<br>rabbitmq-3.9.29<br>rabbitmq-4.0.9<br>rabbitmq-4.1.6<br>rabbitmq-4.2.1<br>rabbitmq-4.3.1 | RabbitMQ is a reliable and mature messaging and streaming broker. | xuriwuyun |
 | redis | redis-5.0.12<br>redis-6.2.14<br>redis-6.2.17<br>redis-6.2.18<br>redis-6.2.19<br>redis-6.2.22<br>redis-7.0.6<br>redis-7.2.10<br>redis-7.2.11<br>redis-7.2.12<br>redis-7.2.14<br>redis-7.2.4<br>redis-7.2.7<br>redis-7.4.2<br>redis-7.4.5<br>redis-7.4.6<br>redis-7.4.7<br>redis-7.4.9<br>redis-8.0.1<br>redis-8.0.3<br>redis-8.0.4<br>redis-8.0.5<br>redis-8.2.1<br>redis-8.2.2<br>redis-8.2.3<br>redis-8.2.6<br>redis-8.4.0<br>redis-8.4.3<br>redis-8.6.3<br>redis-cluster-5.0.12<br>redis-cluster-6.2.14<br>redis-cluster-6.2.17<br>redis-cluster-6.2.18<br>redis-cluster-6.2.19<br>redis-cluster-6.2.22<br>redis-cluster-7.0.6<br>redis-cluster-7.2.10<br>redis-cluster-7.2.11<br>redis-cluster-7.2.12<br>redis-cluster-7.2.14<br>redis-cluster-7.2.4<br>redis-cluster-7.2.7<br>redis-cluster-7.4.2<br>redis-cluster-7.4.5<br>redis-cluster-7.4.6<br>redis-cluster-7.4.7<br>redis-cluster-7.4.9<br>redis-cluster-8.0.1<br>redis-cluster-8.0.3<br>redis-cluster-8.0.4<br>redis-cluster-8.0.5<br>redis-cluster-8.2.1<br>redis-cluster-8.2.2<br>redis-cluster-8.2.3<br>redis-cluster-8.2.6<br>redis-cluster-8.4.0<br>redis-cluster-8.4.3<br>redis-cluster-8.6.3<br>redis-sentinel-5.0.12<br>redis-sentinel-6.2.14<br>redis-sentinel-6.2.17<br>redis-sentinel-6.2.18<br>redis-sentinel-6.2.19<br>redis-sentinel-6.2.22<br>redis-sentinel-7.0.6<br>redis-sentinel-7.2.10<br>redis-sentinel-7.2.11<br>redis-sentinel-7.2.12<br>redis-sentinel-7.2.14<br>redis-sentinel-7.2.4<br>redis-sentinel-7.2.7<br>redis-sentinel-7.4.2<br>redis-sentinel-7.4.5<br>redis-sentinel-7.4.6<br>redis-sentinel-7.4.7<br>redis-sentinel-7.4.9<br>redis-sentinel-8.0.1<br>redis-sentinel-8.0.3<br>redis-sentinel-8.0.4<br>redis-sentinel-8.0.5<br>redis-sentinel-8.2.1<br>redis-sentinel-8.2.2<br>redis-sentinel-8.2.3<br>redis-sentinel-8.2.6<br>redis-sentinel-8.4.0<br>redis-sentinel-8.4.3<br>redis-sentinel-8.6.3<br>redis-twemproxy-0.5.0 | Redis is an in-memory database that persists on disk. The data model is key-value, but many different kind of values are supported: Strings, Lists, Sets, Sorted Sets, Hashes, Streams, HyperLogLogs, Bitmaps. | wangyelei |
 | risingwave | risingwave-compactor-v1.0.0<br>risingwave-compute-v1.0.0<br>risingwave-connector-v1.0.0<br>risingwave-frontend-v1.0.0<br>risingwave-meta-v1.0.0 | RisingWave is a distributed SQL streaming database that enables cost-efficient and reliable processing of streaming data. | RisingWave Labs |

addons-cluster/qdrant/templates/cluster.yaml

@@ -16,3 +16,9 @@ spec:
       replicas: {{ .Values.replicas | default 1 }}
       {{- include "kblib.componentResources" . | indent 6 }}
       {{- include "kblib.componentStorages" . | indent 6 }}
+      configs:
+      - name: qdrant-config-template
+        variables:
+          {{ if .Values.config.default_replication_factor }}
+          default_replication_factor: "{{ .Values.config.default_replication_factor }}"
+          {{- end }}

addons-cluster/qdrant/values.yaml

@@ -25,6 +25,10 @@ memory: 2
 ##
 storage: 20
 
+config:
+  # if not empty, will be set to qdrant's config's storage.collection.replication_factor
+  default_replication_factor: ""
+
 ## customized default values to override kblib chart's values
 extra:
   terminationPolicy: Delete

addons/qdrant/configs/config.yaml.tpl

@@ -100,6 +100,60 @@ storage:
     # Custom M param for hnsw graph built for payload index. If not set, default M will be used.
     payload_m: null
 
+  # Default parameters for collections
+  collection:
+    # Number of replicas of each shard that network tries to maintain
+    {{ if index . "default_replication_factor" }}
+    replication_factor: {{ int .default_replication_factor }}
+    {{ else }}
+      {{ if gt (int .COMP_REPLICA_COUNT) 1 }}
+    replication_factor: 2
+      {{ else }}
+    replication_factor: 1
+      {{ end }}
+    {{ end }}
+
+    # How many replicas should apply the operation for us to consider it successful
+    #write_consistency_factor: 1
+
+    # Default parameters for vectors.
+    #vectors:
+      # Memory placement of the vector storage: cold or cached.
+      # Overrides the deprecated `on_disk` flag if both are set.
+      # memory: null
+
+    # shard_number_per_node: 1
+
+    # Default quantization configuration.
+    # More info: https://qdrant.tech/documentation/manage-data/quantization
+    quantization: null
+
+    # Default strict mode parameters for newly created collections.
+    #strict_mode:
+      # Whether strict mode is enabled for a collection or not.
+      #enabled: false
+
+      # Max allowed `limit` parameter for all APIs that don't have their own max limit.
+      #max_query_limit: null
+
+      # Max allowed `timeout` parameter.
+      #max_timeout: null
+
+      # Allow usage of unindexed fields in retrieval based (eg. search) filters.
+      #unindexed_filtering_retrieve: null
+
+      # Allow usage of unindexed fields in filtered updates (eg. delete by payload).
+      #unindexed_filtering_update: null
+
+      # Max HNSW value allowed in search parameters.
+      #search_max_hnsw_ef: null
+
+      # Whether exact search is allowed or not.
+      #search_allow_exact: null
+
+      # Max oversampling value allowed in search.
+      #search_max_oversampling: null
+
 service:
 
   # Maximum size of POST data in a single request in megabytes

addons/qdrant/scripts-ut-spec/qdrant_restore_spec.sh

@@ -0,0 +1,51 @@
+# shellcheck shell=bash
+
+Describe "Qdrant Restore Bash Script Tests"
+  It "retries after a transient snapshot transport failure"
+    restore_tmp_dir="$(mktemp -d)"
+    cat > "${restore_tmp_dir}/common.sh" <<'COMMON'
+qdrant_set_tls_variables() {
+  SCHEME=http
+  CURL_TLS=""
+}
+
+datasafed() {
+  case "$1:$2" in
+    list:/) printf '%s\n' pod-0 ;;
+    list:/pod-0/) printf '%s\n' /pod-0/test.snapshot ;;
+    pull:*) printf '%s\n' snapshot-data ;;
+  esac
+}
+
+qdrant_curl() {
+  cat >/dev/null
+  count_file="${RESTORE_TEST_TMP}/attempts"
+  attempt=0
+  [ ! -f "$count_file" ] || attempt="$(cat "$count_file")"
+  attempt=$((attempt + 1))
+  printf '%s\n' "$attempt" > "$count_file"
+  if [ "$attempt" -eq 1 ]; then
+    printf '%s\n' 'transient upload failure'
+    return 22
+  fi
+  printf '%s\n' '{"status":"ok"}'
+}
+
+sleep() { :; }
+COMMON
+
+    When run env \
+      RESTORE_TEST_TMP="$restore_tmp_dir" \
+      QDRANT_COMMON_FILE="${restore_tmp_dir}/common.sh" \
+      DP_DATASAFED_BIN_PATH="${restore_tmp_dir}" \
+      DP_BACKUP_BASE_PATH="${restore_tmp_dir}/backup/pod-0" \
+      DP_DB_HOST="qdrant" \
+      bash ../scripts/qdrant-restore.sh
+    The status should be success
+    The output should include "failed to restore collection test in pod-0, retry"
+    The output should include "restore collection test in pod-0 successfully"
+    The contents of file "${restore_tmp_dir}/attempts" should eq "2"
+
+    rm -rf "$restore_tmp_dir"
+  End
+End

addons/qdrant/scripts-ut-spec/qdrant_setup_spec.sh

@@ -89,30 +89,25 @@ Describe "Qdrant Setup Bash Script Tests"
       When call qdrant_start_mode "http://qdrant-provision-qdrant-qdrant.default.svc.cluster.local:6333"
       The status should be success
       The output should eq "bootstrap"
-      The contents of file "$QDRANT_STORAGE_PATH/.kubeblocks-bootstrap-owner" should eq "bootstrap-attempt:pod-uid-initial"
     End
 
     It "does not repeat bootstrap after a container restart of the same pod"
       CURRENT_POD_NAME="qdrant-provision-qdrant-0"
-      BOOTSTRAP_SERVICE_AVAILABLE=false
+      BOOTSTRAP_SERVICE_AVAILABLE=true
       QDRANT_BOOTSTRAP_SERVICE_DISCOVERY_ATTEMPTS=1
-      printf 'bootstrap-attempt:%s\n' "$CURRENT_POD_UID" > "$QDRANT_STORAGE_PATH/.kubeblocks-bootstrap-owner"
       When call qdrant_start_mode "http://qdrant-provision-qdrant-qdrant.default.svc.cluster.local:6333"
       The status should be success
       The output should eq "join"
-      The stderr should include "initial bootstrap was already claimed"
     End
 
     It "does not bootstrap a recreated ordinal zero during a service outage"
       CURRENT_POD_NAME="qdrant-provision-qdrant-0"
       CURRENT_POD_UID="pod-uid-recreated"
-      BOOTSTRAP_SERVICE_AVAILABLE=false
+      BOOTSTRAP_SERVICE_AVAILABLE=true
       QDRANT_BOOTSTRAP_SERVICE_DISCOVERY_ATTEMPTS=1
-      printf '%s\n' "bootstrap-attempt:pod-uid-original" > "$QDRANT_STORAGE_PATH/.kubeblocks-bootstrap-owner"
       When call qdrant_start_mode "http://qdrant-provision-qdrant-qdrant.default.svc.cluster.local:6333"
       The status should be success
       The output should eq "join"
-      The stderr should include "initial bootstrap was already claimed"
     End
 
     It "fails closed when the pod UID needed for an initial bootstrap claim is missing"
@@ -137,17 +132,6 @@ Describe "Qdrant Setup Bash Script Tests"
       The stderr should include "storage is not empty"
     End
 
-    It "fails closed when the initial bootstrap claim cannot be persisted"
-      CURRENT_POD_NAME="qdrant-provision-qdrant-0"
-      QDRANT_BOOTSTRAP_OWNER_FILE="$QDRANT_STORAGE_PATH/missing/owner"
-      BOOTSTRAP_SERVICE_AVAILABLE=false
-      QDRANT_BOOTSTRAP_SERVICE_DISCOVERY_ATTEMPTS=1
-      When call qdrant_start_mode "http://qdrant-provision-qdrant-qdrant.default.svc.cluster.local:6333"
-      The status should be success
-      The output should eq "join"
-      The stderr should include "cannot persist the initial bootstrap claim"
-    End
-
     It "joins when a reused empty ordinal zero observes an existing service during discovery"
       CURRENT_POD_NAME="qdrant-rollout-replace-qdrant-0"
       BOOTSTRAP_SERVICE_AVAILABLE=false
@@ -157,7 +141,6 @@ Describe "Qdrant Setup Bash Script Tests"
       When call qdrant_start_mode "http://qdrant-rollout-replace-qdrant-qdrant.default.svc.cluster.local:6333"
       The status should be success
       The output should eq "join"
-      The contents of file "$QDRANT_STORAGE_PATH/.kubeblocks-bootstrap-owner" should eq "existing-cluster"
     End
 
     It "joins an existing cluster even when KubeBlocks reuses ordinal zero"
@@ -166,27 +149,6 @@ Describe "Qdrant Setup Bash Script Tests"
       When call qdrant_start_mode "http://qdrant-rollout-replace-qdrant-qdrant.default.svc.cluster.local:6333"
       The status should be success
       The output should eq "join"
-      The contents of file "$QDRANT_STORAGE_PATH/.kubeblocks-bootstrap-owner" should eq "existing-cluster"
-    End
-
-    It "fails closed when an observed existing cluster cannot be recorded"
-      CURRENT_POD_NAME="qdrant-rollout-replace-qdrant-0"
-      QDRANT_BOOTSTRAP_OWNER_FILE="$QDRANT_STORAGE_PATH/missing/owner"
-      BOOTSTRAP_SERVICE_AVAILABLE=true
-      When call qdrant_start_mode "http://qdrant-rollout-replace-qdrant-qdrant.default.svc.cluster.local:6333"
-      The status should be failure
-      The output should eq ""
-      The stderr should include "cannot persist the existing-cluster bootstrap marker"
-    End
-
-    It "restarts from durable raft state instead of rejoining"
-      CURRENT_POD_NAME="qdrant-provision-qdrant-0"
-      BOOTSTRAP_SERVICE_AVAILABLE=true
-      echo '{"peer_id":1}' > "$QDRANT_STORAGE_PATH/raft_state.json"
-      When call qdrant_start_mode "http://qdrant-provision-qdrant-qdrant.default.svc.cluster.local:6333"
-      The status should be success
-      The output should eq "restart"
-      The contents of file "$QDRANT_STORAGE_PATH/.kubeblocks-bootstrap-owner" should eq "existing-cluster"
     End
 
     It "waits to join for non-zero pods when the bootstrap service is not ready"

addons/qdrant/scripts/qdrant-backup.sh

@@ -29,15 +29,10 @@ function save_backup_size() {
     echo "{\"totalSize\":\"$TOTAL_SIZE\"}" >"${DP_BACKUP_INFO_FILE}"
 }
 
-if [ "${TLS_ENABLED:-}" = "true" ]; then
-  endpoint=https://${DP_DB_HOST}:6333
-  CURL_TLS="-k"
-else
-  endpoint=http://${DP_DB_HOST}:6333
-  CURL_TLS=""
-fi
+qdrant_set_tls_variables
+endpoint="$SCHEME://${DP_DB_HOST}:6333"
 
-collectionRes=$(qdrant_curl -sS -f ${endpoint}/collections)
+collectionRes=$(qdrant_curl --retry 3 -sS -f ${endpoint}/collections)
 if ! echo "${collectionRes}" | jq -e . >/dev/null 2>&1; then
   echo "failed to parse collections response: ${collectionRes}"
   exit 1
@@ -50,7 +45,7 @@ fi
 # snapshot all collections
 for c in ${collections}; do
   echo "INFO: start to snapshot collection ${c}..."
-  snapshot=$(qdrant_curl -sS -f -XPOST ${endpoint}/collections/${c}/snapshots)
+  snapshot=$(qdrant_curl --retry 3 -sS -f -XPOST ${endpoint}/collections/${c}/snapshots)
   status=$(echo "${snapshot}" | jq -r '.status // empty')
   if [ "${status}" != "ok" ]; then
     echo "backup failed, status: ${status}"
@@ -62,8 +57,8 @@ for c in ${collections}; do
     echo "backup failed, snapshot name is empty"
     exit 1
   fi
-  qdrant_curl -sS -f -L ${endpoint}/collections/${c}/snapshots/${name} | datasafed push - "/${c}.snapshot"
-  qdrant_curl -sS -f -XDELETE ${endpoint}/collections/${c}/snapshots/${name} >/dev/null
+  qdrant_curl --retry 3 -sS -f -L ${endpoint}/collections/${c}/snapshots/${name} | datasafed push - "/${c}.snapshot"
+  qdrant_curl --retry 3 -sS -f -XDELETE ${endpoint}/collections/${c}/snapshots/${name} >/dev/null
   echo "INFO: snapshot collection ${c} successfully."
 done
 save_backup_size

addons/qdrant/scripts/qdrant-common.sh

@@ -110,6 +110,16 @@ qdrant_effective_api_key() {
   qdrant_config_service_api_key
 }
 
+qdrant_set_tls_variables() {
+  if [ "${TLS_ENABLED:-}" = "true" ]; then
+    SCHEME="https"
+    CURL_TLS="-k"
+  else
+    SCHEME="http"
+    CURL_TLS=""
+  fi
+}
+
 # Run curl against the Qdrant API, attaching the api-key header when configured.
 # Temporarily disables xtrace while the command runs so the key is not logged.
 # Args: passed through to curl (URL, -sf, --max-time, etc.)
@@ -126,8 +136,10 @@ qdrant_curl() {
 
   api_key="$(qdrant_effective_api_key)"
   if [ -n "$api_key" ]; then
+    # shellcheck disable=SC2086
     "${QDRANT_CURL_BIN:-curl}" ${CURL_TLS:-} -H "api-key: ${api_key}" "$@"
   else
+    # shellcheck disable=SC2086
     "${QDRANT_CURL_BIN:-curl}" ${CURL_TLS:-} "$@"
   fi
   qdrant_curl_rc=$?

addons/qdrant/scripts/qdrant-member-leave.sh

@@ -578,13 +578,7 @@ qdrant_configure_client() {
   export QDRANT_CURL_BIN
   JQ="${JQ:-/qdrant/tools/jq}"
 
-  if [ "${TLS_ENABLED:-}" = "true" ]; then
-    SCHEME="https"
-    CURL_TLS="-k"
-  else
-    SCHEME="http"
-    CURL_TLS=""
-  fi
+  qdrant_set_tls_variables
 }
 
 qdrant_select_control_endpoint() {

addons/qdrant/scripts/qdrant-restore.sh

@@ -3,40 +3,44 @@
 set -e
 set -o pipefail
 export PATH="$PATH:$DP_DATASAFED_BIN_PATH"
-export DATASAFED_BACKEND_BASE_PATH="$DP_BACKUP_BASE_PATH"
+DATASAFED_BACKEND_BASE_PATH="$(dirname "$DP_BACKUP_BASE_PATH")"
+export DATASAFED_BACKEND_BASE_PATH
 
 QDRANT_COMMON_FILE="${QDRANT_COMMON_FILE:-/qdrant/scripts/common.sh}"
 if [ -r "$QDRANT_COMMON_FILE" ]; then
   # shellcheck disable=SC1090
   . "$QDRANT_COMMON_FILE"
 fi
 
-if [ "${TLS_ENABLED:-}" = "true" ]; then
-  QDRANT_SCHEME=https
-  CURL_TLS="-k"
-else
-  QDRANT_SCHEME=http
-  CURL_TLS=""
-fi
-
-for snapshot in $(datasafed list /) ; do
-  collection_name=${snapshot%.*}
-  # skip file kubeblocks-backup.json which is not a snapshot
-  if [ "${collection_name}" == "kubeblocks-backup" ]; then
-    continue
-  fi
-  echo "INFO: start to restore collection ${collection_name}..."
+qdrant_set_tls_variables
 
-  while true; do
-    if datasafed pull "${snapshot}" - | qdrant_curl -sS -f -X POST "${QDRANT_SCHEME}://${DP_DB_HOST}:6333/collections/${collection_name}/snapshots/upload?priority=snapshot" \
-      -H 'Content-Type:multipart/form-data' \
-      -F "snapshot=@-;filename=${snapshot}" > /tmp/qdrant-restore.log 2>&1 \
-      && grep -q '"status":"ok"' /tmp/qdrant-restore.log; then
-      echo "restore collection ${collection_name} successfully"
-      break
-    else
-      echo "INFO: failed to restore collection ${collection_name}, retry..."
-      sleep 5
+for pod in $(datasafed list /); do
+  for file in $(datasafed list "/$pod/") ; do
+    snapshot=$(basename "$file")
+    collection_name=${snapshot%.*}
+    # skip file kubeblocks-backup.json which is not a snapshot
+    if [ "${collection_name}" == "kubeblocks-backup" ]; then
+      continue
     fi
+    echo "INFO: start to restore collection ${collection_name} in ${pod}..."
+
+    while true; do
+      if datasafed pull "${file}" - | qdrant_curl --retry 3 -sS -f -X POST "${SCHEME}://${DP_DB_HOST}:6333/collections/${collection_name}/snapshots/upload?priority=snapshot" \
+        -H 'Content-Type:multipart/form-data' \
+        -F "snapshot=@-;filename=${snapshot}" > /tmp/qdrant-restore.log 2>&1; then
+        cat /tmp/qdrant-restore.log
+        echo ""
+        if grep -q '"status":"ok"' /tmp/qdrant-restore.log; then
+          echo "restore collection ${collection_name} in ${pod} successfully"
+          break
+        else
+          echo "INFO: failed to restore collection ${collection_name} in ${pod}, retry..."
+          sleep 5
+        fi
+      else
+          echo "INFO: failed to restore collection ${collection_name} in ${pod}, retry..."
+          sleep 5
+      fi
+    done
   done
 done