diff --git a/skills/algolia-crawler/SKILL.md b/skills/algolia-crawler/SKILL.md index 1436843..f56b3c0 100644 --- a/skills/algolia-crawler/SKILL.md +++ b/skills/algolia-crawler/SKILL.md @@ -56,7 +56,7 @@ Follow these steps in order. Full commands, config, and code live in the referen 1. **Set up the CLI and credentials.** Install the CLI and export your Crawler credentials (a Crawler *user ID* + *API key*, distinct from your app keys), plus an Algolia **write** API key for the target index and your **App ID**. See [cli.md](references/cli.md#setup-and-credentials). 2. **Inspect the page first — is it JavaScript-rendered?** Many modern pages load their real content via XHR after load, so a naive crawl indexes empty shells. Detect this and enable rendering. See [javascript-rendering.md](references/javascript-rendering.md). -3. **Write a `recordExtractor` that emits RAG records** following the mental model above — one record per unit, a prose `content` field, structured attributes, and chunking for long prose. See [record-extractor.md](references/record-extractor.md). +3. **Write a `recordExtractor` that emits RAG records.** For docs and prose, the recommended default is Algolia's Markdown helpers (`helpers.markdown` + `helpers.splitTextIntoRecords`) — they preserve headings/lists/code and are Algolia's own AskAI/RAG pattern. Hand-roll an extractor when you need per-entity structured records (tables, catalogs). Either way, follow the mental model above. See [record-extractor.md](references/record-extractor.md). 4. **Validate with `algolia crawler test` BEFORE indexing.** It runs your config against a live URL and returns the records it *would* create, without writing anything. Use it as your feedback loop — and as a DOM inspector to fix selectors against the real rendered markup. See [workflow.md](references/workflow.md#validate-before-you-index). 5. **Apply index settings explicitly.** Do not rely on `initialIndexSettings` to configure the index — in practice it frequently does not apply. Set `searchableAttributes`, `attributesForFaceting`, etc. yourself with `algolia settings import` after the first crawl. See [rag-index-settings.md](references/rag-index-settings.md). 6. **Reindex, then verify** by searching the index (`algolia search`) for a few realistic RAG questions before trusting it. diff --git a/skills/algolia-crawler/references/rag-index-settings.md b/skills/algolia-crawler/references/rag-index-settings.md index e70f334..d0c8388 100644 --- a/skills/algolia-crawler/references/rag-index-settings.md +++ b/skills/algolia-crawler/references/rag-index-settings.md @@ -26,6 +26,8 @@ Why these choices for RAG: - **`customRanking: ["asc(order)"]`.** The `order` you set in the extractor gives a stable, meaningful tiebreak (rank, document order) when text relevance ties. - **`distinct: 0`.** RAG wants recall — keep every chunk. Only raise it if near-duplicate records hurt. +**Using the Markdown helper?** If you index with `helpers.markdown` + `helpers.splitTextIntoRecords` (see [record-extractor.md](record-extractor.md)), the chunk lives in a **`text`** attribute — put `text` first in `searchableAttributes` in place of `content`, and snippet/highlight `text`. + ## Add semantic retrieval (NeuralSearch / vectors) Keyword search alone misses paraphrased questions ("how do I cancel" vs. a doc titled "Ending your subscription"). For RAG, enable Algolia's **NeuralSearch** on the index (dashboard → index → NeuralSearch, or via the API) so retrieval is hybrid keyword + semantic over your `content` field. This is usually the biggest single lever on retrieval quality — recommend it whenever the user's queries will be natural-language questions. diff --git a/skills/algolia-crawler/references/record-extractor.md b/skills/algolia-crawler/references/record-extractor.md index d1aea7e..775eaf5 100644 --- a/skills/algolia-crawler/references/record-extractor.md +++ b/skills/algolia-crawler/references/record-extractor.md @@ -4,9 +4,44 @@ The `recordExtractor` is a JavaScript function the Crawler runs on each page. It In the crawler config the function is stored as a string (`{"__type":"function","source":"…"}`) — that's what `algolia crawler create -F config.json` reads. **The source runs as plain JavaScript — no TypeScript.** Strip any type annotations (`: string[]`, `as any`); they are syntax errors at runtime. -## Emit one record per retrieval unit +## Markdown for RAG — the default for docs and prose -Don't return one record per page. Return one per *idea* — a section, a Q&A, a table row/entity. Use a `record_type` discriminator so you can filter by kind later, and give every record a natural-language `content` field plus clean structured attributes. +For prose content — documentation, articles, guides, marketing pages — the best records are usually **Markdown**, not flattened text. Markdown keeps the structure an LLM benefits from (headings, lists, tables, code blocks, links) in a compact form, and it's the pattern Algolia recommends for AskAI/RAG ([Markdown indexing guide](https://www.algolia.com/doc/guides/algolia-ai/askai/guides/markdown-indexing)). Two built-in helpers do the work: + +- `helpers.markdown("")` — converts the matched HTML subtree to a Markdown string. Target the article body and exclude chrome (nav, header, breadcrumb, the "on this page" TOC). +- `helpers.splitTextIntoRecords({ text, baseRecord, maxRecordBytes, orderingAttributeName })` — splits that Markdown into one or more records under the byte cap. Each record carries the `baseRecord` fields, the chunk in a **`text`** attribute, and a numeric part (via `orderingAttributeName`). ObjectIDs are auto-suffixed per chunk (`#0`, `#1`, …). + +```js +({ url, $, helpers }) => { + const text = helpers.markdown("main [class*=content]"); // ← TUNE this selector to the site + if (!text) return []; + const title = $("h1").first().text().trim() || $("title").text().trim(); + return helpers.splitTextIntoRecords({ + text, + baseRecord: { + record_type: "doc", + url: url.href, + objectID: url.href, + page_title: title, + lang: $("html").attr("lang") || "en", + }, + maxRecordBytes: 4000, // smaller = tighter chunks (better recall); larger = more context per hit, more tokens + orderingAttributeName: "part", + }); +} +``` + +**The selector still needs the sample-and-test step.** Don't trust a generic selector — the docs' own example `main > *:not(nav)…` can grab only the "On this page" TOC on a given site. Use `crawler test` to check which selector captures the real article body (try `main`, `article`, `main [class*=content]`, …) before trusting it. Same discover → sample → test loop as everywhere else. + +**Tuning `maxRecordBytes`.** Smaller chunks (a few thousand bytes) retrieve more precisely and keep the LLM context lean; larger chunks give more context per hit but cost more tokens per answer. Start around 3000–5000 for RAG and adjust. + +When you use this pattern the Markdown lives in the **`text`** attribute — make `text` your primary `searchableAttribute` (see [rag-index-settings.md](rag-index-settings.md)). + +## Emit one record per retrieval unit (for structured data) + +When the page is **structured data** — a table, leaderboard, pricing grid, catalog — hand-roll an extractor instead, so you get one record per entity with clean, typed attributes rather than a wall of prose. (For docs/prose, prefer the Markdown helper above.) + +Return one record per *idea* — an entity/row, a Q&A. Use a `record_type` discriminator so you can filter by kind later, and give every record a natural-language `content` field plus clean structured attributes. ```js ({ url, $ }) => {