Using the commit method on the Document class removes previous data from the associated IndexedDB

[cbissonnier]

I am using React.js and Typescript, using the [email protected].

I am creating a Document with an IndexedDB, and the index on it works as expected, I get the data I want on page load (data being in the indexedDB).
My issue is whenever I add new documents to the index, and then use the commit method, it inserts the new data but removes all the existing ones, instead of adding new ones.

Here are the parts of my code where I perform those actions:

function createFlexSearchIndexes(): FlexSearch.Document {
  // Create the Document index
  return new FlexSearch.Document({
    doc: {
      id: 'id',
      index: ['message', 'type', 'flatTags', 'flatAttributes'],
      store: [
        'id',
        'eventTimestamp',
        'type',
        'message',
        'flatAttributes',
        'flatTags',
        'receivedTimestamp',
        'attributes',
        'tags',
        'severity',
      ],
    },
    tokenize: 'forward',
    cache: 1000,
  })
}

async function createOrLoadFlexSearchDB(): Promise<{
  index: FlexSearch.Document
}> {
  console.time('dbLoad')

  const index = createFlexSearchIndexes()
  const db = new IndexedDB('my-storage')

  await index.mount(db)

  console.timeEnd('dbLoad')
  return { index }
}

Initialization of the flexsearch document:

useEffect(() => {
    const initDB = async () => {
      try {
        if (!indexRef.current) {
          const { index } = await createOrLoadFlexSearchDB()
          indexRef.current = index


          const allDocs = await index.search('test', {
            query: '',
            limit: 10000,
            enrich: true,
            field: 'message',
          })

          setSearchResults(
            allDocs[0].result.map((doc) => ({
              id: doc.id,
              eventTimestamp: doc.doc.eventTimestamp,
              receivedTimesamp: doc.doc.receivedTimestamp,
              type: doc.doc.type,
              message: doc.doc.message,
              severity: doc.doc.severity,
              attributes: JSON.parse(doc.doc.attributes),
              tags: JSON.parse(doc.doc.tags),
            }))
          )

        }
      } catch (error) {
        console.error('Failed to initialize FlexSearch:', error)
      }
    }
    initDB()
  }, [])

and here is the code snippet where I try to add more data:

const setData = useCallback(
    async (data) => {
      if (!indexRef.current) {
        console.log('FlexSearch not initialized')
        return
      }

      for (const doc of data) {
        indexRef.current.add(doc.id, {
          id: doc.id, 
          eventTimestamp: doc.eventTimestamp,
          type: doc.type,
          message: doc.message || '',
          flatAttributes: flattenAttributes(
            doc.attributes as Record<string, unknown>
          ),
          flatTags: flattenTags(doc.tags as Record<string, unknown>),
          receivedTimestamp: doc.receivedTimestamp,
          attributes: JSON.stringify(doc.attributes || {}),
          tags: JSON.stringify(doc.tags || {}),
          severity: doc.severity || 9,
        })
      }

      try {
        await indexRef.current.commit(false)

        setSearchResults((prev) => [...prev, ...data])
      } catch (error) {
        console.error('Error committing to index:', error)
      }
    },
    []
  )

Note that I am setting the param replace_all_contents to false to ensure the the indexedDB content does not get replaced. Let me know if you need more info, thank you for looking into this.

[ts-thomas]

Thanks for the report. The latest release should fix that issue.