Skip to content

Commit d61d4c4

Browse files
masonwyatt23claude
andcommitted
feat(core,cli): provider plugin marketplace & hot-reload system
- Add PluginManifest type (provider-plugin-manifest.ts): id, name, version, search(), findSimilar?, auth, capabilities, healthCheckUrl, validatePluginManifest() - Add loadPluginFromPath(path) and watchPluginDirectory(dir) for runtime discovery and hot-reload (provider-plugin-loader.ts) - Export new symbols from packages/core/src/index.ts - Integrate with existing providerRegistry.register() via registerProvider() - CLI surface: webfetch plugin list|add|test in packages/cli/src/commands.ts - Comprehensive test suite (tests/plugins.test.ts): 74 tests covering manifest validation, fake plugin injection, version conflict resolution, auth isolation, watchPluginDirectory initial scan + reload, bootstrapPluginDirectory, CLI surface, registry integration, capability propagation, and security checks - Plugin development guide at docs/PLUGIN_DEVELOPMENT.md with template skeleton Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent b8b4f1c commit d61d4c4

6 files changed

Lines changed: 2321 additions & 2 deletions

File tree

docs/PLUGIN_DEVELOPMENT.md

Lines changed: 319 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,319 @@
1+
# Plugin Development Guide
2+
3+
Build and ship a custom image-source provider for webfetch without touching core.
4+
5+
## Overview
6+
7+
The webfetch plugin system lets you add third-party image sources — Google Images scrapers, Etsy, ArtStation, custom enterprise image databases — at runtime via a standard `PluginManifest` interface.
8+
9+
Plugins are standalone TypeScript/JavaScript modules that export a `manifest` constant. The host discovers them by path, validates the manifest schema, and wires the plugin into federation automatically.
10+
11+
## Quick Start
12+
13+
### 1. Create your plugin file
14+
15+
```ts
16+
// my-plugin/index.ts
17+
import type { PluginManifest } from "webfetch-core";
18+
19+
export const manifest: PluginManifest = {
20+
id: "my-image-source", // unique machine ID, no spaces
21+
name: "My Image Source", // display name
22+
version: "1.0.0", // semver
23+
capabilities: ["search"],
24+
25+
// Required: the search function
26+
search: async (query, opts) => {
27+
const resp = await fetch(`https://api.myimagesource.com/search?q=${encodeURIComponent(query)}`);
28+
const data = await resp.json();
29+
return data.images.map((img: any) => ({
30+
url: img.url,
31+
source: "my-image-source",
32+
license: "CC0",
33+
title: img.title,
34+
}));
35+
},
36+
37+
// Optional: metadata
38+
description: "Search images from My Image Source",
39+
healthCheckUrl: "https://api.myimagesource.com/health",
40+
defaultLicense: "CC0",
41+
auth: {
42+
envVars: ["MY_IMAGE_SOURCE_API_KEY"],
43+
},
44+
};
45+
```
46+
47+
### 2. Load it
48+
49+
**One-shot (CLI):**
50+
```sh
51+
webfetch plugin add ./my-plugin/index.ts
52+
```
53+
54+
**Programmatic:**
55+
```ts
56+
import { loadPluginFromPath } from "webfetch-core";
57+
58+
const result = await loadPluginFromPath("./my-plugin/index.ts");
59+
if (!result.success) throw new Error(result.error);
60+
console.log("Loaded:", result.manifest.id);
61+
```
62+
63+
**Directory watch (hot-reload):**
64+
```ts
65+
import { watchPluginDirectory } from "webfetch-core";
66+
67+
const watcher = await watchPluginDirectory("./plugins", {
68+
onLoad: (r) => console.log(r.success ? `Loaded ${r.manifest?.id}` : `Failed: ${r.error}`),
69+
onUnload: (id) => console.log(`Unloaded ${id}`),
70+
});
71+
72+
// Later:
73+
watcher.stop();
74+
```
75+
76+
### 3. Verify it
77+
78+
```sh
79+
webfetch plugin list
80+
webfetch plugin test my-image-source
81+
webfetch plugin test my-image-source --query "sunset beach" --json
82+
```
83+
84+
---
85+
86+
## PluginManifest Reference
87+
88+
| Field | Type | Required | Description |
89+
|-------|------|----------|-------------|
90+
| `id` | `string` || Unique machine ID. No whitespace. |
91+
| `name` | `string` || Human-readable display name. |
92+
| `version` | `string` || Semver (`"1.0.0"`). |
93+
| `capabilities` | `PluginCapability[]` || At minimum `["search"]`. |
94+
| `search` | `function` || `(query: string, opts: SearchOptions) => Promise<ImageCandidate[]>` |
95+
| `findSimilar` | `function` || `(ref, opts) => Promise<ImageCandidate[]>`. Auto-adds `"findSimilar"` capability. |
96+
| `description` | `string` || Short description (≤200 chars). |
97+
| `healthCheckUrl` | `string` || URL for health checks (`webfetch plugin test`). |
98+
| `defaultLicense` | `string` || Default license tag (e.g. `"CC0"`, `"CC_BY"`). |
99+
| `auth` | `PluginAuth` || Auth contract (see below). |
100+
| `icon` | `string` || URL/data-URI of a provider icon. |
101+
| `coreVersion` | `string` || Semver range of webfetch-core this plugin targets. |
102+
| `author` | `string` || Author name/email. |
103+
| `repository` | `string` || Plugin repository URL. |
104+
105+
### PluginAuth
106+
107+
```ts
108+
interface PluginAuth {
109+
envVars?: string[]; // env vars read by the plugin (e.g. ["MY_API_KEY"])
110+
keys?: string[]; // SearchOptions.auth fields used (e.g. ["myApiKey"])
111+
scopes?: string[]; // Human-readable permission scopes (for display)
112+
}
113+
```
114+
115+
---
116+
117+
## Template: Full-Featured Provider
118+
119+
```ts
120+
// artstation-plugin/index.ts
121+
import type { PluginManifest, ImageCandidate, SearchOptions } from "webfetch-core";
122+
123+
const BASE_URL = "https://www.artstation.com/api/v2";
124+
125+
async function search(query: string, opts: SearchOptions): Promise<ImageCandidate[]> {
126+
const apiKey = opts.auth?.artstationApiKey ?? process.env.ARTSTATION_API_KEY;
127+
if (!apiKey) {
128+
console.warn("artstation: ARTSTATION_API_KEY not set — skipping");
129+
return [];
130+
}
131+
132+
const signal = opts.signal ?? AbortSignal.timeout(opts.timeoutMs ?? 15_000);
133+
const url = `${BASE_URL}/search?q=${encodeURIComponent(query)}&type=artwork&page=1&per_page=${opts.maxPerProvider ?? 20}`;
134+
135+
const resp = await fetch(url, {
136+
headers: { Authorization: `Bearer ${apiKey}` },
137+
signal,
138+
});
139+
140+
if (!resp.ok) {
141+
throw new Error(`artstation: HTTP ${resp.status}`);
142+
}
143+
144+
const data = await resp.json();
145+
return (data.data ?? []).map((item: any) => ({
146+
url: item.cover_url,
147+
thumbnailUrl: item.smaller_square_cover_url,
148+
source: "artstation",
149+
license: "EDITORIAL_LICENSED",
150+
title: item.title,
151+
author: item.user?.username,
152+
sourcePageUrl: `https://www.artstation.com/artwork/${item.hash_id}`,
153+
width: item.cover_asset?.width,
154+
height: item.cover_asset?.height,
155+
} satisfies ImageCandidate));
156+
}
157+
158+
export const manifest: PluginManifest = {
159+
id: "artstation",
160+
name: "ArtStation",
161+
version: "1.0.0",
162+
capabilities: ["search"],
163+
description: "Professional digital artwork and concept art from ArtStation.",
164+
healthCheckUrl: `${BASE_URL}/projects.json?page=1`,
165+
defaultLicense: "EDITORIAL_LICENSED",
166+
auth: {
167+
envVars: ["ARTSTATION_API_KEY"],
168+
},
169+
author: "Your Name <you@example.com>",
170+
repository: "https://github.com/your-org/webfetch-artstation",
171+
search,
172+
};
173+
```
174+
175+
---
176+
177+
## Template: Plugin with Reverse-Image Search
178+
179+
```ts
180+
import type { PluginManifest, ImageCandidate, SearchOptions } from "webfetch-core";
181+
182+
async function search(query: string, opts: SearchOptions): Promise<ImageCandidate[]> {
183+
// ... your search implementation
184+
return [];
185+
}
186+
187+
async function findSimilar(
188+
ref: { url?: string; bytes?: Uint8Array },
189+
opts: SearchOptions,
190+
): Promise<ImageCandidate[]> {
191+
const imageUrl = ref.url ?? "<upload bytes to get URL>";
192+
// ... call your reverse-image API
193+
return [];
194+
}
195+
196+
export const manifest: PluginManifest = {
197+
id: "my-reverse-search",
198+
name: "My Reverse Search",
199+
version: "1.0.0",
200+
// "findSimilar" is auto-added when findSimilar function is present
201+
capabilities: ["search"],
202+
search,
203+
findSimilar, // ← makes this provider available via findByCapability("findSimilar")
204+
healthCheckUrl: "https://api.example.com/health",
205+
};
206+
```
207+
208+
---
209+
210+
## Directory Layout
211+
212+
The watcher expects one of these layouts:
213+
214+
```
215+
plugins/
216+
my-plugin.plugin.ts # flat file: must end in .plugin.ts/.js/.mjs
217+
another-plugin.plugin.ts
218+
219+
artstation/ # subdirectory: first match wins
220+
index.ts # ← loaded as entry point
221+
helpers.ts
222+
223+
google-images/
224+
plugin.ts # ← also works
225+
```
226+
227+
---
228+
229+
## CLI Commands
230+
231+
| Command | Description |
232+
|---------|-------------|
233+
| `webfetch plugin list` | List all registered plugin providers |
234+
| `webfetch plugin list --json` | JSON output for scripting |
235+
| `webfetch plugin add <path>` | Load a plugin from a local path |
236+
| `webfetch plugin test <id>` | Smoke-test: health check + search("test") |
237+
| `webfetch plugin test <id> --query "sunset"` | Test with custom query |
238+
| `webfetch plugin test <id> --json` | JSON output for assertions |
239+
240+
---
241+
242+
## Programmatic API
243+
244+
```ts
245+
import {
246+
loadPluginFromPath, // load single plugin by path
247+
bootstrapPluginDirectory, // one-shot bulk load from directory
248+
watchPluginDirectory, // load + watch for hot-reload
249+
validatePluginManifest, // validate manifest shape without loading
250+
listPluginProviders, // list all registered plugin descriptors
251+
registerProvider, // low-level: register a ProviderPluginDescriptor
252+
unregisterPluginProvider, // remove a plugin from registry
253+
} from "webfetch-core";
254+
```
255+
256+
### Integration at bootstrap
257+
258+
```ts
259+
import { bootstrapRegistry, bootstrapPluginDirectory } from "webfetch-core";
260+
261+
// 1. Register built-in providers (idempotent)
262+
bootstrapRegistry();
263+
264+
// 2. Load community plugins from a directory
265+
const pluginDir = process.env.WEBFETCH_PLUGINS_DIR ?? path.join(os.homedir(), ".webfetch/plugins");
266+
const results = await bootstrapPluginDirectory(pluginDir, { replace: false });
267+
268+
const failed = results.filter((r) => !r.success);
269+
if (failed.length > 0) {
270+
console.warn("Plugin load failures:", failed.map((r) => `${r.path}: ${r.error}`));
271+
}
272+
```
273+
274+
---
275+
276+
## Publishing a Plugin
277+
278+
1. **npm package** — name it `webfetch-plugin-<name>` by convention.
279+
2. **Export `manifest`** as a named export from the package entry point.
280+
3. **Declare `peerDependency`** on `webfetch-core`.
281+
4. **Set `coreVersion`** in the manifest to the range you tested against.
282+
283+
```json
284+
{
285+
"name": "webfetch-plugin-artstation",
286+
"version": "1.0.0",
287+
"main": "dist/index.js",
288+
"types": "dist/index.d.ts",
289+
"peerDependencies": {
290+
"webfetch-core": ">=0.1.0"
291+
}
292+
}
293+
```
294+
295+
Users install and load it:
296+
297+
```sh
298+
npm install webfetch-plugin-artstation
299+
webfetch plugin add ./node_modules/webfetch-plugin-artstation
300+
```
301+
302+
Or programmatically:
303+
304+
```ts
305+
import { loadPluginFromPath } from "webfetch-core";
306+
await loadPluginFromPath(require.resolve("webfetch-plugin-artstation"));
307+
```
308+
309+
---
310+
311+
## Best Practices
312+
313+
- **Respect `opts.timeoutMs`** — attach an `AbortSignal` so federation can cancel slow requests.
314+
- **Respect `opts.maxPerProvider`** — don't return more results than requested.
315+
- **Return `source: "<your-id>"`** on every `ImageCandidate` so attribution works.
316+
- **Set `healthCheckUrl`** — enables `webfetch plugin test` and circuit-breaker health checks.
317+
- **Handle missing auth gracefully** — return `[]` with a `console.warn` rather than throwing.
318+
- **Declare `auth.envVars`** — lets users discover what keys are needed via `webfetch plugin list`.
319+
- **Use semver** for `version` — enables conflict detection when multiple plugin versions are installed.

0 commit comments

Comments
 (0)