Control vectors vs LoRA adapters

[campis]

I'm struggling to understand the differences between these two techniques and need some guidance on how to apply them effectively. My ultimate goal is to create custom persona cards and integrate them into an already fine-tuned model. Since both techniques seem to be well-suited for this purpose, which one would be the best choice?

[jukofyork]

In mathematical terms, control vectors just add a single direction to the output regardless:

y = A*x + c

whereas LoRAs take the input and transform it:

y = A*x + B*x

Also control vectors only act on the output of a particular tensor in the LLM (called down_proj), whereas LoRAs can act on any tensor.

In general control vectors are really good at affecting a single really well defined change to the model. For example my 8 creative writing "axis":

https://huggingface.co/jukofyork/creative-writing-control-vectors-v3.0

seem to work well for almost any model.

BUT: If you want to affect more than a single well defined change, they don't work at all well (and looking at the mathematics it's clear why - they can only add that single vector direction!).

A case in point when I first started working on the creative writing control vectors; I tried for several weeks to get the 2-axis system from AD&D working: good/evil and law/chaos... It was very easy to get the good/evil axis working, but the law/chaos was basically impossible due to it being so "multifaceted".

---

So I think your use case:

My ultimate goal is to create custom persona cards and integrate them into an already fine-tuned model.

Is really more suited to LoRAs, but you could also try to split up your "persona" into a set of axis and see if you can then use vector addition to blend the persona axis.

[campis]

Thank you for such a detailed reply!