Adding serializer and api implementation for segment profiler state #2687

oaganesh · 2025-04-30T04:27:17Z

Description

Adding implementation for profiling and statistical analysis of KNN vector segments in OpenSearch with compression when benchmarking. Provides functionality to collect, process, and store statistical information about vector dimensions across different shards and segments.

Related Issues

Implements #2622

Check List

[✔️ ] New functionality includes testing.
[✔️ ] New functionality has been documented.
[✔️ ] API changes companion pull request created.
[✔️ ] Commits are signed per the DCO using --signoff.
[✔️ ] Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check ✔️.

Example input:

curl -XPUT "http://localhost:9200/target_index" -H 'Content-Type: application/json' -d'{      
  "settings": {
    "index": {
      "knn": true,
      "number_of_shards": 1,
      "number_of_replicas": 0
    }
  },                                                                                            
  "mappings": {
    "properties": {
      "target_field": {
        "type": "knn_vector",
        "dimension": 128,
        "space_type": "l2",
        "compression_level": "16x",
        "method": {
          "name": "hnsw",
          "space_type": "l2",
          "engine": "faiss",
          "parameters": {
            "ef_construction": 100,
            "m": 16
          }
        }
      }
    }
  }
}'

curl -XPOST "http://localhost:9200/target_index/_doc" -H 'Content-Type: application/json' -d'{
  "target_field": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 1.0, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2.0]
}'

curl -X GET "localhost:9200/_plugins/_knn/warmup/target_index?pretty"

curl -X GET "localhost:9200/_plugins/_knn/profile/target_index?pretty"

Example output:

  "total_shards" : 1,
  "successful_shards" : 1,
  "failed_shards" : 0,
  "profile_results" : {
    "target_index" : {
      "dimensions" : [
        {
          "min" : 1.3,
          "max" : 1.9,
          "variance" : 0.0,
          "mean" : 3.8,
          "count" : 5,
          "std_deviation" : 0.0,
          "sum" : 1.9,
          "dimension" : 0
        },
        {
          "min" : 0.2,
          "max" : 1.89
          "variance" : 0.2,
          "mean" : 3.78,
          "count" : 5,
          "std_deviation" : 0.3,
          "sum" : 1.89,
          "dimension" : 1
        }
      ]
    }
  }
}

Signed-off-by: Arun Ganesh <[email protected]>

markwu-sde · 2025-04-30T07:53:46Z

CHANGELOG.md

 ## [Unreleased 2.x](https://github.com/opensearch-project/k-NN/compare/2.19...2.x)
 ### Features
+* [Vector Profiler] Adding basic generic vector profiler implementation and tests. [#2624](https://github.com/opensearch-project/k-NN/pull/2624)
+* [Vector Profiler] Updating serializer, compression, and api. [#2687](https://github.com/opensearch-project/k-NN/pull/2687)


Should we be a little more specific on what we're updating?

Also AFAIK we're not updating anything related to how compression is done or any API (but we're introducing one). Should we reword this?

markwu-sde · 2025-04-30T07:53:50Z

src/main/java/org/opensearch/knn/index/KNNIndexShard.java

+     * @param fieldName The name of the vector field to profile
+     * @return List of statistical summaries for each dimension
+     */
+    // TODO: Write unit tests to ensure that the segment statistic aggregation is correct.


Should we finish this TODO and write unit tests for the aggregation?

markwu-sde · 2025-04-30T07:53:51Z

src/main/java/org/opensearch/knn/index/KNNIndexShard.java

+     * @return List of statistical summaries for each dimension
+     */
+    // TODO: Write unit tests to ensure that the segment statistic aggregation is correct.
+    public List<StatisticalSummaryValues> profile(String fieldName) {


NIT: final variables whenever possible

markwu-sde · 2025-04-30T07:53:53Z

src/main/java/org/opensearch/knn/index/KNNIndexShard.java

+
+            // Get dimension and validate all segments have the same dimension
+            int dimension = segmentLevelProfilerStates.get(0).getDimension();
+            boolean dimensionsMatch = segmentLevelProfilerStates.stream().allMatch(state -> state.getDimension() == dimension);


Would there ever be a case where the dimensions won't match accordingly here? Do we need to check for this?

Reverting back to previous way of implementing KNNIndexShard as current way is less robust.

markwu-sde · 2025-04-30T07:53:54Z

src/main/java/org/opensearch/knn/index/KNNIndexShard.java

+
+            if (!dimensionsMatch) {
+                log.error("[KNN] Inconsistent dimensions found across segments");
+                return shardVectorProfile; // Return empty list


Nit: you can return List.of()

markwu-sde · 2025-04-30T07:53:55Z

src/main/java/org/opensearch/knn/index/KNNIndexShard.java

+
+                for (SegmentProfilerState state : segmentLevelProfilerStates) {
+                    List<SummaryStatistics> stateStats = state.getStatistics();
+                    if (dimensionId < stateStats.size()) {


Again - if the dimension is constant here do we really need this check? it will always be within bounds.

Understood, reverting back to previous way of implementing KNNIndexShard as current way is less robust.

markwu-sde · 2025-04-30T07:53:56Z

src/main/java/org/opensearch/knn/index/KNNIndexShard.java

+                    List<SummaryStatistics> stateStats = state.getStatistics();
+                    if (dimensionId < stateStats.size()) {
+                        SummaryStatistics stat = stateStats.get(dimensionId);
+                        if (stat != null) {


When can this be possible?

markwu-sde · 2025-04-30T07:53:59Z

src/main/java/org/opensearch/knn/index/KNNIndexShard.java

+            log.info("[KNN] Profiling completed for field: {} in shard: {}", fieldName, indexShard.shardId());
+        } catch (Exception e) {
+            log.error(
+                "[KNN] Critical error during profiling for field: {} in shard: {}: {}",


Critical sounds a bit too extreme here given we're not in the hot path.

markwu-sde · 2025-04-30T07:54:02Z

src/main/java/org/opensearch/knn/index/KNNIndexShard.java

+     * @return List of statistical summaries for each dimension
+     */
+    public List<StatisticalSummaryValues> profile() {
+        return profile("target_field");


We shouldn't hard code a field name here. I understand we're doing this from a benchmarking OSB perspective but if we want customers to be able to leverage this feature we need to provide the ability to take in a field/index name as input.

Removed specific field input

markwu-sde · 2025-04-30T07:54:03Z

src/main/java/org/opensearch/knn/index/codec/KNN990Codec/KNN990QuantizationStateReader.java

        int fieldNumber = segmentReadState.fieldInfos.fieldInfo(field).getFieldNumber();

-        try (IndexInput input = segmentReadState.directory.openInput(quantizationStateFileName, IOContext.READONCE)) {
+        try (IndexInput input = segmentReadState.directory.openInput(quantizationStateFileName, IOContext.DEFAULT)) {


Could you give some context around this? Why are we changing from READONCE to DEFAULT?

Mostly did it for consistency when accessing quantization state files multiple times.
https://lucene.apache.org/core/9_12_1/core/org/apache/lucene/store/IOContext.html?is-external=true

markwu-sde · 2025-04-30T07:54:07Z

src/main/java/org/opensearch/knn/index/codec/KNN990Codec/NativeEngines990KnnVectorsWriter.java

            // should skip graph building only for non quantization use case and if threshold is met
            if (quantizationState == null && shouldSkipBuildingVectorDataStructure(totalLiveDocs)) {
-                log.debug(
+                log.info(


NIT: use log.debug wherever possible

markwu-sde · 2025-04-30T07:54:11Z

src/main/java/org/opensearch/knn/index/codec/KNN990Codec/NativeEngines990KnnVectorsWriter.java

+
+        SegmentProfilerState segmentProfilerState = null;
+        if (totalLiveDocs > 0) {
+            // TODO:Refactor to another init


This seems important to the PR. Are we going to act on this TODO?

Refactors NativeEnginesVectorsWriter class to encompass segmentStateWriter and initSegmentStateWriterIfNecessary()

markwu-sde · 2025-04-30T07:54:13Z

src/main/java/org/opensearch/knn/index/query/SegmentLevelQuantizationUtil.java

+    /**
+     * A utility function to get {@link SegmentProfilerState} for a given segment and field.
+     * This needs to public as we are accessing this on a transport action
+     * TODO: move this out of this Util class and into another one.


Same here. This seems important to the PR. Should we complete this TODO?

The method has already been refactors into it's own class for under org.opensearch.knn.index.query. Need to remove the existing code.

markwu-sde

One over-arching comment here is that we should have unit/integration tests for all our code. Make sure to include them in the PR so that we can test our functionality such as making sure the aggregation at the shard level is what we're expecting.

markwu-sde · 2025-04-30T07:55:00Z

src/main/java/org/opensearch/knn/index/query/SegmentLevelQuantizationUtil.java

+     * @return {@link SegmentProfilerState}
+     * @throws IOException exception during reading the {@link SegmentProfilerState}
+     */
+    public static SegmentProfilerState getSegmentProfileState(final LeafReader leafReader, String fieldName) throws IOException {


We have the util function below that is a exact replica of this method. Do we still need this?

No, the method has already been refactors into it's own class for under org.opensearch.knn.index.query

markwu-sde · 2025-04-30T07:55:11Z

src/main/java/org/opensearch/knn/plugin/transport/KNNProfileResponse.java

+    /**
+     * Get aggregated dimension statistics by index
+     */
+    private Map<String, Map<Integer, Map<String, Object>>> getAggregatedStats() {


I'm assuming you want to get the aggregation at the field level here across all shards.

Shouldn't this be by field/index instead of just index?

NIT: getAggregatedStats can be more specific because there are now multiple layers of aggregation

markwu-sde · 2025-04-30T07:55:14Z

src/main/java/org/opensearch/knn/plugin/transport/KNNProfileResponse.java

+                continue;
+            }
+
+            Map<Integer, Map<String, Object>> dimensions = indexDimensions.computeIfAbsent(indexName, k -> new HashMap<>());


So the mapping that you're trying to do is:

Map<Index, Map<Dimension ID, Map<String,Object>>.

In lucene and specifically within k-nn a index can have multiple fields. Not all fields are of type knn_vector

markwu-sde · 2025-04-30T07:55:15Z

src/main/java/org/opensearch/knn/plugin/transport/KNNProfileResponse.java

+            Map<Integer, Map<String, Object>> dimensions = indexDimensions.computeIfAbsent(indexName, k -> new HashMap<>());
+
+            for (int i = 0; i < stats.size(); i++) {
+                StatisticalSummaryValues stat = stats.get(i);


Please correct me if I'm wrong, but I think the crux of your situation here by looking at StatisticalSummaryValues is that the library we're using doesn't provide us with a aggregation technique for that object.

If that is the case, how about we collect our SegmentProfileStates and put them in the shard response? That way we can craft both the shard/field level aggregation as we see fit since we have access to those values.

markwu-sde · 2025-04-30T07:55:18Z

src/main/java/org/opensearch/knn/plugin/transport/KNNProfileResponse.java

+                int finalI = i;
+                Map<String, Object> dimension = dimensions.computeIfAbsent(i, k -> {
+                    Map<String, Object> newDim = new HashMap<>();
+                    newDim.put("dimension", finalI);


NIT: static imports when possible

markwu-sde · 2025-04-30T07:56:01Z

src/main/java/org/opensearch/knn/plugin/transport/KNNProfileShardResult.java

+            double min = in.readDouble();
+            double max = in.readDouble();
+            double sum = in.readDouble();
+            this.dimensionStats.add(new StatisticalSummaryValues(mean, variance, n, min, max, sum));


If we're deserialize it here again would it be worthwhile to just save the deserialization at the profile API layer?

What kind of responses do we need to deserialize here?

markwu-sde · 2025-04-30T07:56:05Z

src/main/java/org/opensearch/knn/plugin/transport/KNNWarmupTransportAction.java

        );
+
        knnIndexShard.warmup();
+        knnIndexShard.profile();


Do we need this in WarmupTransportAction?

markwu-sde · 2025-04-30T07:56:25Z

src/main/java/org/opensearch/knn/profiler/SegmentProfilerState.java

+            out.writeVInt(dimension);
+            out.writeVInt(statistics.size());
+            for (SummaryStatistics stat : statistics) {
+                out.writeDouble(stat.getMean());


We shouldn't be serializing/deserializing the values but rather the object itself.

SummaryStatistics already implements the serializable interface so we should be able to convert it directly to bytes without needing to retrieve it's properties.

markwu-sde · 2025-04-30T07:56:28Z

src/main/java/org/opensearch/knn/profiler/SegmentProfilerState.java

+
+            for (int i = 0; i < statsSize; i++) {
+                SummaryStatistics stat = new SummaryStatistics();
+                stat.addValue(input.readDouble());


I think we've previously had a discussion on the deserialization aspect of this - the same issue is present here as well.

Let's make sure to modify this as this would yield incorrect results for our profiler.

Yes, this was just used to help test for compression by creating a new instance. Reverting back to pre-existing method for serializing and deserializing.

markwu-sde · 2025-04-30T08:00:00Z

src/main/java/org/opensearch/knn/plugin/transport/KNNProfileResponse.java

+ * Response for KNN profile request
+ */
+@Log4j2
+public class KNNProfileResponse extends BroadcastResponse implements ToXContentObject {


Have not looked too closely into these since I do have some comments on the serialization/deserialization component.

Signed-off-by: Arun Ganesh <[email protected]>

markwu-sde · 2025-05-01T19:57:17Z

src/main/java/org/opensearch/knn/plugin/transport/KNNProfileResponse.java

+            builder.endObject();
+        }
+        builder.endObject();
+


What about at the cluster level with every segment aggregated together? This output currently does this per shard right? Can we combine every SummaryStatistics in the cluster together?

Yes the stats were currently per shard - added cluster level stats as well.

markwu-sde · 2025-05-01T19:57:18Z

src/main/java/org/opensearch/knn/plugin/transport/KNNProfileResponse.java

+                        if (dim < state.getStatistics().size()) {
+                            SummaryStatistics stats = state.getStatistics().get(dim);
+
+                            totalCount += stats.getN();


Should we leverage a existing library for aggregation?

This doesn't give an correct aggregation outside of min max and sum.

https://commons.apache.org/proper/commons-math/commons-math-docs/apidocs/src-html/org/apache/commons/math4/legacy/stat/descriptive/AggregateSummaryStatistics.html#line.305 - there's a library provided to us that aggregates a SummaryStatistics object for us. Can we use that instead of trying to combine it on our own?

Updated to use AggregateSummaryStatistics

markwu-sde · 2025-05-01T19:57:21Z

src/main/java/org/opensearch/knn/plugin/transport/KNNProfileResponse.java

+            builder.startObject(shardProfileResult.shardId);
+
+            // Individual segment statistics
+            builder.startArray("segments");


Can you provide an example of output in the Javadoc of what this would look like?

Updated javadocs

markwu-sde · 2025-05-01T19:57:23Z

src/main/java/org/opensearch/knn/plugin/transport/KNNProfileResponse.java

+
+    @Override
+    public void writeTo(StreamOutput streamOutput) throws IOException {
+        super.writeTo(streamOutput);


Let's see if we use this override for anything. If we don't we can throw an exception here to make sure we don't stray off the intended path

markwu-sde · 2025-05-01T19:57:29Z

src/main/java/org/opensearch/knn/plugin/transport/KNNIndexShardProfileResult.java

+     * Constructor for reading from StreamInput
+     */
+    public KNNIndexShardProfileResult(StreamInput streamInput) throws IOException {
+        this.shardId = streamInput.readString();


Do we ever use this constructor?

No, it was used previously in KNNProfileResponse, but when removing the logic and the above constructor I was able to validate the output was still working.

Signed-off-by: Arun Ganesh <[email protected]>

markwu-sde · 2025-05-02T20:55:26Z

CHANGELOG.md


 ## [Unreleased 2.x](https://github.com/opensearch-project/k-NN/compare/2.19...2.x)
 ### Features
+* [Vector Profiler] Adding basic generic vector profiler implementation and tests. [#2624](https://github.com/opensearch-project/k-NN/pull/2624)


Why are we adding these changes? Weren't they already merged?

Yes, updated branch to adjust with changes already implemented in feature branch

markwu-sde · 2025-05-02T20:55:32Z

src/main/java/org/opensearch/knn/profiler/SegmentProfilerState.java

+ */
+@Log4j2
+@AllArgsConstructor
+public class SegmentProfilerState implements Serializable {


Why is this file being shown as new? Isn't this already in the branch?

Updated branch to adjust with changes already implemented in feature branch

…erializer Signed-off-by: oaganesh <[email protected]>

markwu-sde

Please add unit/integration tests to validate shard aggregation functionality. You'll need to create multiple segments/shards to do this most likely.

Signed-off-by: Arun Ganesh <[email protected]>

jmazanec15

A few comments. I think overall structure is looking good. I think itd be good to enhance tests to ensure that we are validating functionality. Seems you are already working on that!

jmazanec15 · 2025-05-06T23:01:15Z

src/main/java/org/opensearch/knn/plugin/transport/KNNProfileRequest.java

+@Getter
+public class KNNProfileRequest extends BroadcastRequest<KNNProfileRequest> {
+
+    private String index;


Broadcast request should already have the indices: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/action/support/broadcast/BroadcastRequest.java#L52.

jmazanec15 · 2025-05-06T23:01:37Z

src/main/java/org/opensearch/knn/plugin/transport/KNNProfileRequest.java

+public class KNNProfileRequest extends BroadcastRequest<KNNProfileRequest> {
+
+    private String index;
+    private String field;


nit: lets make these final.

jmazanec15 · 2025-05-06T23:02:33Z

src/main/java/org/opensearch/knn/plugin/transport/KNNProfileRequest.java

+
+    @Override
+    public ActionRequestValidationException validate() {
+        return null;


nit: no need to override

jmazanec15 · 2025-05-06T23:05:07Z

src/main/java/org/opensearch/knn/plugin/transport/KNNProfileResponse.java

+ *   "failures": []
+ * }
+ */
+public class KNNProfileResponse extends BroadcastResponse implements ToXContentObject {


nit: BroadcastResponse already implements ToXContentObject - do we need to do it again? https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/action/support/broadcast/BroadcastResponse.java

jmazanec15 · 2025-05-06T23:05:29Z

src/main/java/org/opensearch/knn/plugin/transport/KNNProfileResponse.java

+ */
+public class KNNProfileResponse extends BroadcastResponse implements ToXContentObject {
+
+    List<KNNIndexShardProfileResult> shardProfileResults;


nit: private final?

jmazanec15 · 2025-05-06T23:12:24Z

src/main/java/org/opensearch/knn/plugin/transport/KNNProfileTransportAction.java

+    KNNProfileResponse,
+    KNNIndexShardProfileResult> {
+
+    public static Logger logger = LogManager.getLogger(KNNProfileTransportAction.class);


replace this with @Log4j2 annotation

jmazanec15 · 2025-05-06T23:12:51Z

src/main/java/org/opensearch/knn/plugin/transport/KNNWarmupTransportAction.java

        KNNIndexShard knnIndexShard = new KNNIndexShard(
            indicesService.indexServiceSafe(shardRouting.shardId().getIndex()).getShard(shardRouting.shardId().id())
        );
+


nit: remove for git history

jmazanec15 · 2025-05-06T23:14:34Z

src/main/java/org/opensearch/knn/profiler/SegmentProfileKNNCollector.java

+
+@Setter
+@Getter
+public class SegmentProfileKNNCollector implements KnnCollector {


Add comment on top of this class.

jmazanec15 · 2025-05-06T23:17:18Z

src/test/java/org/opensearch/knn/plugin/transport/KNNProfileTransportActionIT.java

+        return response;
+    }
+
+    private void validateProfileResponse(Response response, int dimension) throws IOException, ParseException {


Itd be good to update this to ensure expected statistics are being computed. For instance, you can generate the vectors from a normalized distribution (i.e. 0-1) and confirm that the mean values is computed as close to 0.5.

+1 agree with this.

Make sure to validate that the aggregation is what'd we expect. Since merge is out of scope for this this can probably be done be explicitly creating the 2 segments and calling the profile API to make sure the aggregate is what we expect.

Updated tests

jmazanec15 · 2025-05-06T23:18:16Z

src/main/java/org/opensearch/knn/plugin/transport/KNNIndexShardProfileResult.java

+import java.util.List;
+
+@AllArgsConstructor
+public class KNNIndexShardProfileResult implements Writeable {


Do we also need deserialization logic with this?

I think it may be useful especially when compressing vectors.

Signed-off-by: Arun Ganesh <[email protected]>

Updating serializer, compression, and api.

33ad02f

Signed-off-by: Arun Ganesh <[email protected]>

oaganesh requested review from 0ctopus13prime, VijayanB, Vikasht34, heemin32, jmazanec15, junqiu-lei, luyuncheng, martin-gaievski, naveentatikonda, navneet1v, ryanbogan, shatejas and vamshin as code owners April 30, 2025 04:27

oaganesh changed the base branch from main to feature/vector-profile April 30, 2025 04:27

Updating changelog and formatting.

5cce54e

Signed-off-by: Arun Ganesh <[email protected]>

markwu-sde reviewed Apr 30, 2025

View reviewed changes

oaganesh changed the title ~~Updating serializer, compression, and api.~~ Adding serializer and api implementation for segment profiler state Apr 30, 2025

Updating aggregation and serializer for api implementation.

b4a80e1

Signed-off-by: Arun Ganesh <[email protected]>

markwu-sde reviewed May 1, 2025

View reviewed changes

Updating aggregation and refactoring constructors.

5b2fbda

Signed-off-by: Arun Ganesh <[email protected]>

markwu-sde reviewed May 2, 2025

View reviewed changes

Merge branch 'feature/vector-profile' into main-compression-segment-s…

40375eb

…erializer Signed-off-by: oaganesh <[email protected]>

markwu-sde reviewed May 3, 2025

View reviewed changes

This was referenced May 4, 2025

Test compression segment serializer #2696

Closed

Adding main segment implementation for API and indexing. #2653

Closed

oaganesh added 2 commits May 5, 2025 16:24

Adding unit and integration tests for shard aggregation.

4a6d45b

Signed-off-by: Arun Ganesh <[email protected]>

Updating testing configuration.

6464751

Signed-off-by: Arun Ganesh <[email protected]>

jmazanec15 reviewed May 6, 2025

View reviewed changes

Refactoring tests and serialization for API.

3f9bc09

Signed-off-by: Arun Ganesh <[email protected]>

oaganesh closed this Jun 17, 2025

Adding serializer and api implementation for segment profiler state #2687

Adding serializer and api implementation for segment profiler state #2687

Uh oh!

Conversation

oaganesh commented Apr 30, 2025

Description

Related Issues

Check List

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

markwu-sde Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

markwu-sde Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

markwu-sde left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

markwu-sde Apr 30, 2025 •

edited

Loading

markwu-sde Apr 30, 2025 •

edited

Loading

oaganesh May 1, 2025 •

edited

Loading