fastest way to create and store embeddings for data in a SQLite table

[punkish]

I have a SQLite table like so (simplified schema below) with ~950K rows

CREATE TABLE t (id INTEGER PRIMARY KEY, fulltext TEXT);

What is the quickest and easiest way to generate embeddings and store them in a vectors table? I am using Ollama with llama 3.2 running locally using the nomic-embed-text embeddings model. I would like to do this once and then create a web interface to query the data. Additionally, as new rows get added to the table t, I would like to TRIGGER embeddings for them as well. Is that possible? If yes, any hint would be very welcome.

[punkish]

Is there a way to use transactions when inserting a lot of embeddings in a libSQL db? Right now I am using a pattern like so, and it is really slow (I have about a million documents for which I need to insert embeddings). I'd like to insert them in transactions of 5000 at a time

const rows = db.prepare(`SELECT fulltext FROM t LIMIT 5000`).all();

for (const row of result.rows) {
    await app.addLoader(new TextLoader({ text: row.fulltext }));
}

[github-actions]

This issue is stale because it has been open for 14 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 30 days since being marked as stale.

[github-actions]

This issue is stale because it has been open for 14 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 30 days since being marked as stale.