Points within training data can trigger drift

[MSymeono]

Description

When checking the cosine similarity of an I/O within the training data set, with K Means turned off, sometimes drift will register. Without any abstraction, a point from within the training data should never measure anything but a cosine similarity of 1.
Possibilities as we see it in order of likelihood:

1. Embedding inaccuracy: The python vs javascript embedding process may not be exactly 1:1, meaning that even though we are using the "same" model, the tokenizers or other settings behind the curtains might be causing the same idea to be embedded differently.
   Solution: Embed all entries (training data AND one off) using exactly the same embedding process, so we can be sure that embeddings will match one another.
2. Model inaccuracy: It may be that the embeddings are fine, but that the models themselves are inaccurate, or are being fed data in a different manner than they would expect. There are many potential models that could slot in here as a replacement, but we didn't have the time to find the perfect model.
   Solution: Evaluate different models and find the best one(s) for the job at hand.
3. HNSW Inaccuracy: Unlikely, but it is possible that the models have embedded perfectly, and our current HNSW constructions and library are unable to find an exact 1:1 match within the training data.
   Solution: More extensive HNSW testing and potentially reviewing other HNSW libraries could yield a more accurate search. Note: "more accurate" usually means more of a time cost in the individual runs, and we have tested with ef_construction and ef_search being set to 300, with M being set to 30, (ef_c is usually 200, ef_s is usually 50, and m is usually 12-16, for reference) and were still unable to have a cosine similarity of 1. During this testing, our HNSW searches took seconds longer than with a more normal set of HNSW values. Because of this, we don't think this is the issue, but we could be wrong.

Reproduction

Dev/Master (any branch)
Reproduction steps

1. Clear existing data folder.
2. Bulk load training data (turn off K-means if your data size is over 100,000 entries)
3. Run a one-off embedding with a point that you know is within your data.
4. compare cosine similarity of the one off and the training data. Expected result: Cosine similarity of 1, as the point definitely should exist within the data
   Actual result: Cosine similarity of <1, indicating some amount of drift.

System information

System information for the two devices we used to test this:
System:
OS: Linux 5.15 Ubuntu 22.04.5 LTS 22.04.5 LTS (Jammy Jellyfish)
CPU: (32) x64 13th Gen Intel(R) Core(TM) i9-13900KF
Memory: 9.81 GB / 15.53 GB
Container: Yes
Shell: 5.1.16 - /bin/bash
Binaries:
Node: 22.13.1 - ~/.nvm/versions/node/v22.13.1/bin/node
npm: 11.3.0 - ~/.nvm/versions/node/v22.13.1/bin/npm
Browsers:
Chrome: 133.0.6943.126

  System:
    OS: Linux 5.15 Ubuntu 22.04.5 LTS 22.04.5 LTS (Jammy Jellyfish)
    CPU: (12) x64 AMD Ryzen 5 5600X 6-Core Processor
    Memory: 6.27 GB / 7.72 GB
    Container: Yes
    Shell: 5.1.16 - /bin/bash
  Binaries:
    Node: 22.14.0 - ~/.nvm/versions/node/v22.14.0/bin/node
    npm: 10.9.2 - ~/.nvm/versions/node/v22.14.0/bin/npm

### Additional information

_No response_

### 👨‍👧‍👦 Contributing

- [ ] 🙋‍♂️ Yes, I'd love to make a PR to fix this bug!