Skip to content

LLM as text encoder implementation from LLM2CLIP#42

Merged
vdplasthijs merged 8 commits intodevelopfrom
feature/llm2vec
Feb 4, 2026
Merged

LLM as text encoder implementation from LLM2CLIP#42
vdplasthijs merged 8 commits intodevelopfrom
feature/llm2vec

Conversation

@gabrieletijunaityte
Copy link
Contributor

@gabrieletijunaityte gabrieletijunaityte commented Feb 4, 2026

What does this PR do?

  • Normalisation check moved to init
  • Cosine similarity fixed not to normalise twice.
  • The implementation of LLM as a text encoder from LLM2CLIP.

I had to do a lot of work arounds to make it work due to dependency issues and unclear environment recommendations from LLM2LCIP. Right now, it uses my forked LLM2vec repo and LLAMA customised version to get rid of flash-atto and spda.
What remains is to unify/decide on model dtypes and potentially adapt LLM2CLIP vision branch as eo encoder.

Before submitting

  • Did you make sure title is self-explanatory and the description concisely explains the PR?
  • Did you make sure your PR does only one thing, instead of bundling different changes together?
  • Did you list all the breaking changes introduced by this pull request?
  • Did you test your PR locally with pytest command?

Copy link
Collaborator

@vdplasthijs vdplasthijs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, curious to see the results! I'll merge this, and we can change the normalisation in a new PR? Thanks!

@vdplasthijs vdplasthijs merged commit 9ed2c26 into develop Feb 4, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants