WebWorker bulk add to reduce postMessages pressure on main thread

[boardend]

We are tweaking the indexing performance of some rather large worker indexes with flexsearch 0.7.0. (500'000+ documents).

I would like to test if adding multiple documents together could reduce the postMessage overhead on the main JS thread. But the flexsearch API doesn't provide a bulk way of adding many documents together. So my question is if you have ever thought about it, or event tested this out? If not, how hard could it be to test/implement such a bulk add in flexsearch?

At the moment we are adding the documents in chunks and do a small setTimeout in between to let some time for the UI to render. So it is a tradeoff between time the indexing will take and letting enough time for the main JS thread to not block the rendering of the app.

[ts-thomas]

I will do some blocking test. For this reason there was originally a setTimeout placed in background. Because pushing to the event loop stack is less blocking than pushing to the message channel. But also when adding as Bulks, you'll need to await in between transmission, because putting larger chunks through the message channel (incl. serialization) is a very performance intense task.

[brunohaid]

As someone looking at a similar situation: why not passing the data as an Arraybuffer, and then let the worker thread do it's thing without interfering with the main JS thread?