Search does not support partial string matching

[calycekr]

It seems that the search doesn't support partial string matching unless the entire word is typed in fully, no results are returned.

[nsarrazin]

Yeah unfortunately this is a limitation of using the mongodb $text index, I think. It handles plurals and some very basic stemming but that's it.

I'm curious if someone knows a good alternative solution but I'm not sure if there's one, it needs be:

• Done entirely with mongodb (no elasticsearch or equivalent embedded into chat-ui)
• Support multiple instances of chat-ui running at once
• Compatible with very large DB sizes (Our conversations collection is on the order of hundreds of GBs...)

I guess that's asking quite a lot 😅 But open to suggestions here!

[eunjikangi]

	`let convs = await collections.conversations
		.find({
			sessionId: undefined,
			...authCondition(locals),
			$text: { $search: searchQuery },
		})
		.sort({
			updatedAt: -1, // Sort by date updated in descending order
		})
		.project({
			title: 1,
			updatedAt: 1,
			model: 1,
			assistantId: 1,
			messages: 1,
			userId: 1,
		})
		.skip(p * 5)
		.limit(5)
		.toArray();

	if (convs.length === 0) {
		console.log("No results found with text search, falling back to regex search");

		convs = await collections.conversations
			.find({
				sessionId: undefined,
				...authCondition(locals),
				"messages.content": { $regex: searchQuery, $options: "i" },
			})
			.sort({
				updatedAt: -1, // Sort by date updated in descending order
			})
			.project({
				title: 1,
				updatedAt: 1,
				model: 1,
				assistantId: 1,
				messages: 1,
				userId: 1,
			})
			.skip(p * 5)
			.limit(5)
			.toArray();
	}

	const processedConvs = convs.map((conv) => {
		...
	});

	return Response.json(processedConvs as GETSearchEndpointReturn);
}`

I initially performed a check using a $text, and if no results were found, I implemented a fallback to re-evaluate the substring using regular expressions (regex).