Skip to content

Commit 347bc4e

Browse files
lxfu1福晋
lxfu1
and
福晋
authored
feat: add transformer implementation (#1)
* feat: add transformer implementation * feat: bind the cache to embedder * fix: build error * chore: add git ignore * chore: remove sample embedder * chore: remove temp placeholder absolute paths * chore: remove chunk * chore: add utils * chore: context opt * chore: remove memory store * chore: remove registry * chore: update readme * chore: update hook timeout --------- Co-authored-by: 福晋 <liufu.lf@antgroup.com>
1 parent 3670846 commit 347bc4e

39 files changed

Lines changed: 3323 additions & 300 deletions

‎.eslintrc.json‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
{
2+
"env": {
3+
"node": true,
4+
"es2020": true
5+
},
6+
"parserOptions": {
7+
"ecmaVersion": 2020,
8+
"sourceType": "module"
9+
},
10+
"parser": "@typescript-eslint/parser",
11+
"extends": [
12+
"eslint:recommended",
13+
"plugin:@typescript-eslint/recommended",
14+
"prettier"
15+
],
16+
"plugins": ["@typescript-eslint"],
17+
"rules": {
18+
"@typescript-eslint/no-explicit-any": "warn",
19+
"@typescript-eslint/no-unused-vars": ["warn", { "argsIgnorePattern": "^_" }],
20+
"@typescript-eslint/explicit-function-return-type": "off",
21+
"@typescript-eslint/no-non-null-assertion": "warn",
22+
"no-console": "off"
23+
},
24+
"ignorePatterns": ["dist/", "node_modules/", "test/fixtures/"]
25+
}

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,3 +11,5 @@ package-lock.json
1111
yarn.lock
1212
pnpm-lock.yaml
1313
pnpm-workspace.yaml
14+
.test-tmp-*
15+
.vscode

‎.prettierrc.json‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"semi": true,
3+
"singleQuote": true,
4+
"trailingComma": "all",
5+
"printWidth": 100,
6+
"tabWidth": 2,
7+
"arrowParens": "always"
8+
}

‎README.md‎

Lines changed: 145 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -3,15 +3,15 @@
33
A local context retrieval library that enables semantic search over your documentation. It loads documents (Markdown, JSON, Text), vectorizes them using [Transformers.js](https://huggingface.co/transformers.js), and stores vectors locally in `.zvec` files for fast semantic querying.
44

55
> [!TIP]
6-
> Base it, We provide an official context HTTP server simlar with context7, used to provide AI code generation context services in MCP, Skill, and CLI, for free!
6+
> Based on this library, we provide an official context HTTP server similar to context7, used to provide AI code generation context services in MCP, Skill, and CLI, for free!
77
88

99
## Features
1010

11-
- 📄 **Multi-format Support**: Supports Markdown, JSON, Text, and other file formats
12-
- 📚 **Multi-library Support**: Manage documents by library
13-
- ⚡ **Auto-indexing**: Automatic vectorization on load
14-
- 🔍 **Semantic Retrieval**: Retrieve relevant documents based on vector similarity (file-level)
11+
- 📄 **Multi-format Support**: Markdown, JSON, Text 文档自动加载与向量化
12+
- 🔍 **Hybrid Retrieval**: 向量语义 + FTS 全文检索双路召回，RRF 融合排序
13+
- 🔁 **Two-stage Reranking**: KeywordReranker 精排，关键词命中优先
14+
- 🌐 **Query Expansion**: 用户自定义同义词表，CN↔EN 跨语言召回增强
1515

1616

1717
## Quick Start
@@ -23,48 +23,132 @@ npm install @antv/context
2323
```typescript
2424
import { Context } from '@antv/context';
2525

26+
// Standard creation — specify vectorsDir
2627
const ctx = await Context.create({ vectorsDir: './vectors' });
2728

2829
// Load documents into a specific library with automatic vectorization
2930
await ctx.load('g2', './g2-docs/**/*.md');
3031
await ctx.load('f2', './f2-docs/**/*.json');
3132

32-
// Query
33+
// Query a library (default: hybrid search + reranking)
3334
const results = await ctx.query('How to configure a line chart', { library: 'g2', topK: 5 });
34-
// => [{ content: '...', score: 0.92, id: 'g2-docs/line.md' }, ...]
35+
// => [{ content: '...', score: 0.92, scoreMode: 'reranked', id: 'g2-docs/line.md' }, ...]
36+
37+
// Close when done (releases resources)
38+
await ctx.close();
3539
```
3640

3741

3842
## API
3943

4044
### `Context.create(options)`
4145

42-
| Parameter | Type | Description |
43-
|-----------|------|-------------|
44-
| `vectorsDir` | `string` | Directory to store vector files |
45-
| `model` | `string` | Transformers model name, default `sentence-transformers/all-MiniLM-L6-v2` |
46+
| Parameter | Type | Default | Description |
47+
|-----------|------|---------|-------------|
48+
| `vectorsDir` | `string` | — | **Required**. Directory to store vector files |
49+
| `basePath` | `string` | `process.cwd()` | Base path for resolving document IDs. Set for cross-machine consistent IDs. |
50+
| `onProgress` | `(phase, detail) => void` | — | Progress callback for `load()` phases: `'load'` → `'embed'` → `'insert'`. |
51+
| `queryExpansion` | `QueryExpansionOptions | false` | `false` (no-op) | Query expansion with user-provided synonym map. `false` disables. Without `synonyms`, expansion is a no-op. |
52+
| `ftsFields` | `string[]` | `['content']` | Fields to index for Full Text Search in hybrid mode |
53+
| `ftsFieldWeights` | `Record<string, number>` | `{ content: 1 }` | Per-field boost weights for FTS text path. Higher = more influence. |
54+
| `rankConstant` | `number` | `60` | RRF rank constant for hybrid search fusion. Lower = "winner-takes-all", higher = more even. |
4655

47-
### `ctx.load(library, glob)`
56+
#### Weight Configuration Example
4857

49-
Load files into a specified library with automatic vectorization. Document ID defaults to the file path.
58+
```typescript
59+
const ctx = await Context.create({
60+
vectorsDir: './vectors',
61+
// Boost title matches 3x over content matches
62+
ftsFieldWeights: { content: 1, title: 3 },
63+
// More "winner-takes-all" ranking
64+
rankConstant: 20,
65+
});
66+
```
67+
68+
#### Query Expansion Configuration Example
69+
70+
```typescript
71+
const ctx = await Context.create({
72+
vectorsDir: './vectors',
73+
// Define your own CN↔EN synonym bridges (no built-in defaults)
74+
queryExpansion: {
75+
synonyms: {
76+
'折线图': ['line chart', '折线'],
77+
'雷达图': ['radar chart', '蜘蛛图'],
78+
'tooltip': ['提示框', 'hover', '悬浮'],
79+
},
80+
},
81+
});
82+
83+
// Disable query expansion entirely
84+
const ctxNoExpand = await Context.create({
85+
vectorsDir: './vectors',
86+
queryExpansion: false,
87+
});
88+
```
89+
90+
### `ctx.load(library, pattern)`
91+
92+
Load files into a specified library with automatic batch vectorization. Documents are embedded in batches and inserted into the vector store. A content-hash change detection mechanism re-embeds files whose content has changed since the last load.
93+
94+
Document IDs are derived from file paths relative to `basePath` for cross-machine consistency.
95+
96+
| Parameter | Type | Description |
97+
|-----------|------|-------------|
98+
| `library` | `string` | Library name for organizing documents |
99+
| `pattern` | `string | string[]` | Glob pattern(s) matching files to load |
50100

51101
```typescript
52102
await ctx.load('g2', './docs/**/*.md');
53103
await ctx.load('g2', ['./docs/**/*.md', './docs/**/*.json']);
54104
```
55105

106+
Load phases emit progress via the `onProgress` callback:
107+
108+
```typescript
109+
const ctx = await Context.create({
110+
vectorsDir: './vectors',
111+
onProgress: (phase, detail) => {
112+
console.log(`${phase}: ${detail.loaded}/${detail.total}`);
113+
},
114+
});
115+
// Phases: 'load' → 'embed' → 'insert'
116+
```
117+
56118
### `ctx.query(text, options)`
57119

58-
Vector similarity retrieval.
120+
Two-stage retrieval: coarse search (vector / hybrid) → reranking → final topK results.
59121

60-
| Parameter | Type | Description |
61-
|-----------|------|-------------|
62-
| `library` | `string` | Required, library to query |
63-
| `topK` | `number` | Number of results to return, default 5 |
122+
| Parameter | Type | Default | Description |
123+
|-----------|------|---------|-------------|
124+
| `library` | `string` | — | Library name to query. |
125+
| `topK` | `number` | `5` | Number of results to return |
126+
127+
```typescript
128+
// Semantic search — hybrid (vector + FTS) + reranking by default
129+
const results = await ctx.query('sankey diagram', { library: 'g2', topK: 5 });
130+
```
131+
132+
#### Query Result Fields
133+
134+
Each result includes:
135+
136+
| Field | Type | Description |
137+
|------|------|-------------|
138+
| `id` | `string` | Document ID |
139+
| `content` | `string` | Document content |
140+
| `score` | `number` | Similarity score (0–1) |
141+
| `scoreMode` | `'vector' | 'hybrid' | 'reranked'` | How the score was computed |
142+
| `meta` | `Record<string, unknown>` | Front-matter metadata (if present) |
143+
| `sourceFilePath` | `string` | Original file path relative to `basePath` |
144+
145+
146+
### `ctx.close()`
147+
148+
Close all stores and release resources. Call this when you are done using the Context instance.
64149

65150
```typescript
66-
const results = await ctx.query('How to configure a line chart', { library: 'g2', topK 5 });
67-
// => [{ id: 'g2-docs/line.md', content: '...', score: 0.92 }, ...]
151+
await ctx.close();
68152
```
69153

70154

@@ -85,20 +169,52 @@ const results = await ctx.query('How to configure a line chart', { library: 'g2'
85169
+--------------+--------------+ |
86170
v v
87171
+-----------------+ +-----------------+
88-
| FileLoader | | Transformers |
89-
+--------+--------+ +--------+--------+
90-
| |
91-
+--------v--------+ |
92-
| Transformers | |
93-
+--------+--------+ |
94-
| |
172+
| FileLoader | | QueryExpander |
173+
+--------+--------+ | (SynonymExpander)|
174+
| +--------+--------+
95175
+--------v--------+ |
96-
| .zvec |<---------------Query-------+
97-
+-----------------+
176+
| EmbedBatch | v
177+
+--------+--------+ +--------v--------+
178+
| | Embedder |
179+
+--------v--------+ +--------+--------+
180+
| .zvec | |
181+
+-----------------+ +--------v--------+
182+
| Vectorize |
183+
+--------+--------+
184+
|
185+
+--------+-----------+
186+
|
187+
+-----------v-----------+
188+
| |
189+
+-------v-------+ +-------v-------+
190+
| FTS Text Path | | Vector Path |
191+
|(ftsFieldWeights| | |
192+
+-------+-------+ +-------+-------+
193+
| |
194+
+-----------+-----------+
195+
|
196+
+-----------v-----------+
197+
| RRF Fusion |
198+
| (rankConstant) |
199+
+-----------+-----------+
200+
|
201+
+-----------v-----------+
202+
| KeywordReranker |
203+
| (optional, 2nd stage) |
204+
+-----------+-----------+
205+
|
206+
Query Result
98207
99208
+------------------------------------------------------------------------+
100209
```
101210

211+
### Module Structure
212+
213+
- **Public API**: `Context`, `QueryOptions`, `QueryResult`, `Document`, `Loader`, `MarkdownLoader`, `JsonLoader`, `TextLoader`, `pathToId`
214+
- **Reranking**: `KeywordReranker`, `createReranker`, `Reranker`, `RerankCandidate`, `RerankResult`, `RerankOptions`
215+
- **Query Expansion**: `SynonymExpander`, `NoopExpander`, `QueryExpander`, `QueryExpansionOptions`
216+
- **Advanced API**: `Embedder`, `TransformersEmbedder`, `EmbedderManager`, `IZvecStore`, `ActualZvecStore`, `Store`
217+
102218

103219
## License
104220

‎package.json‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,12 @@
44
"description": "A local context retrieval library that enables semantic search over your documentation. It loads documents (Markdown, JSON, Text), vectorizes them using [Transformers.js](https://huggingface.co/transformers.js), and stores vectors locally in `.zvec` files for fast semantic querying. ",
55
"main": "dist/index.js",
66
"types": "dist/index.d.ts",
7-
"type": "module",
87
"scripts": {
98
"build": "tsc",
10-
"test": "vitest run --coverage"
9+
"test": "HF_ENDPOINT=https://hf-mirror.com vitest run --coverage",
10+
"lint": "eslint src/ --ext .ts",
11+
"lint:fix": "eslint src/ --ext .ts --fix",
12+
"format": "prettier --write 'src/**/*.ts' 'test/**/*.ts'"
1113
},
1214
"keywords": [
1315
"context",
@@ -28,7 +30,12 @@
2830
},
2931
"devDependencies": {
3032
"@types/node": "^20.0.0",
33+
"@typescript-eslint/eslint-plugin": "^7.0.0",
34+
"@typescript-eslint/parser": "^7.0.0",
3135
"@vitest/coverage-v8": "^3.2.6",
36+
"eslint": "^8.0.0",
37+
"eslint-config-prettier": "^9.0.0",
38+
"prettier": "^3.0.0",
3239
"typescript": "^5.0.0",
3340
"vitest": "^3.2.6"
3441
},

0 commit comments

Comments
 (0)