Add OpenAPI imports and live OAuth integrations

[michielbdejong]

Adds an Integrations section to /app/sync and the atomic-server import-oad CLI command. Clicking GitHub or Google starts OAuth and immediately imports data into the running server after consent, with progress, errors and a link to the imported drive on Sync. API discovery, pagination, ontology derivation and Atomic Data conversion remain in reflector-rs / syncables.

Behavior

• Discovers integration folders under REFLECTOR_ROOT/spec (currently github and google-calendar, displayed as GitHub and Google). Reads each document and its overlays; provider endpoints, scopes, constants and client-credential environment variable names are declarative.
• Current upstream auth overlays are bearer-only, so server/integrations/ supplies supplemental OAuth declarations. A folder's own OAuth flow and metadata take precedence. The generic server host handles browser-bound state, PKCE, the callback, access-token expiry/401 refresh and import status.
• Uses the existing live Db; no second database process or server shutdown is needed for UI imports. Tokens stay in server memory for the import and are never returned to the browser. Pending authorization expires after ten minutes; imports time out after thirty minutes. Reconnecting starts OAuth again; persistent credentials and scheduled sync are not included.
• Requires signed Atomic agent requests to list/start integrations. Imported record namespaces and drive ownership are scoped to that agent, including account-relative Google calendars such as primary. Callbacks are bound to the initiating browser and single-use. Declined authorization reports a failed job.
• Preserves percent-encoded resource paths in HTTP GET handling. Decoding %2F before verifying signatures changed both the signed URL and the identity of imported resources, preventing the returned drive link from opening.
• Retains the CLI workflow for imports with the server stopped, upgrading Reflector to ec3553bf705f2003cab743e4312bb2fee833f11c and adapting to its multi-platform configuration. Both workflows rebuild search indexes after sync, including collected sync errors/partial results.

Setup

Set REFLECTOR_ROOT, GITHUB_CLIENT_ID, GITHUB_CLIENT_SECRET, GOOGLE_CLIENT_ID, and GOOGLE_CLIENT_SECRET when starting AtomicServer. Register http://localhost:9883/integrations/callback for both OAuth applications (use the configured server origin for other deployments). Optional GITHUB_API_CONSTANTS and GOOGLE_CALENDAR_API_CONSTANTS select the source repository/calendar. Buttons indicate missing OAuth setup.

The usage guide covers registration, configuration, token lifetime, partial imports and the CLI. Git dependencies are pinned; the workspace patches Reflector's atomic_lib to this checkout so Storelike/database types agree. Published dependencies are still required before publishing this server version to crates.io.

Validation

Passed:

• Rust build/check and cargo clippy -p atomic-server --no-default-features --features light --test it (existing warnings outside this change).
• CLI importer test: local API/overlay, persistence, repeat updates, owner grants, origin mismatch, API error status.
• Live integration test: real running AtomicServer plus local OAuth/API provider, signed endpoints, PKCE, missing browser cookie, replay rejection, expired access-token refresh, immediate import, drive reads while running, another agent's access denial and cancelled consent.
• All three existing server_cli tests.
• Frontend production build (reusing existing WASM), new component lint, changed UI formatting, Rust formatting and diff checks.
• Manual browser check on an isolated server: Integrations shows GitHub and Google with OAuth-setup-needed buttons.

pnpm typecheck was run and remains blocked by 14 errors in unchanged files: missing Node globals/modules in existing tests, plus two instanceof errors in src/views/Document/upgradeDocument.ts. No errors are reported in the new component. Actual GitHub/Google consent cannot be tested until the operator registers OAuth apps and supplies credentials; the complete protocol/import flow is covered with the local provider.

Related Issues

Closes #1381

Checklist

• Add changelog entry linking to issue, describe API changes
• Add or update tests if needed
• Update docs if needed

Follow-up fixes

• Build downstream workspaces with the same atomic_lib source as Reflector; verified from a separate consumer workspace.
• Use Git CLI fetches for pinned CI dependencies.
• Run live imports outside HTTP workers and bound each provider request to 60 seconds; report failed background tasks instead of leaving them importing.
• Save completed imports into the private-drive list used by My drives.
• Verify sorted drive-child queries and stalled-provider errors in regression tests.

Validation: real-server OAuth integration test, provider-deadline unit test, indexed-prefix unit test, separate-workspace cargo check and production UI build pass. The UI typecheck still reports pre-existing errors in unrelated test and document-migration files.

The accompanying Reflector scan optimization is prepared locally (51 tests and strict Clippy pass) and awaits its repository-required human review before the dependency pin can be updated.

[michielbdejong]

[joepio]

This looks complementary to the work on feat/plugin-model: Reflector/syncables gives us a declarative way to cover many APIs, while that branch adds the integration lifecycle: discovery, connections, reviewed imports, ongoing sync, recovery, and integration actions/events used by automations and the assistant.

I think we should converge on one integration platform with multiple ways to author connectors. A possible path:

1. Keep the OpenAPI + overlay engine. Reuse its API discovery, pagination and ontology derivation instead of rewriting these in each JavaScript connector. Keep the CLI entry point too.
2. Unify the product and connection lifecycle. Expose these connectors in the same integration store, with shared authorization, connection status and scheduling. Consolidate the OAuth implementations behind a common host interface rather than keeping separate setup flows on Sync and Integrations. Managed and self-hosted deployments should use the same implementation with different provider-app configuration.
3. Add an adapter at the storage boundary. For the integrated flow, translate Reflector/syncables records into the common import intents instead of writing directly through AtomicStorage. This should preserve source identity through the existing parent + localId machinery, allow a selected nested destination, and use the same preview, conflict handling and recovery. We should verify how its read/list/delete operations map to that contract; this is more than replacing put.
4. Separate API schemas from shared domain mappings. Deriving an ontology is useful, but a provider's schema is not automatically our shared task/calendar schema. Allow explicit mappings into reusable Atomic properties and templates, preserving provider-specific fields where needed.
5. Make execution guarantees explicit. The current Reflector path is trusted Rust in the server; the plugin branch uses sandboxed JavaScript. We should decide which capabilities and limits the declarative engine needs rather than imply that both have the same isolation. Custom JS can remain an escape hatch for transformations and provider-specific behavior. Bidirectional sync still needs explicit conflict, deletion and uncertain-write semantics beyond OpenAPI.

Google Calendar seems like a good first convergence test: use this PR's discovery/pagination, feed it through the common importer, and expose it in the existing integration store. Start with a reviewed one-way import. Test nested placement, repeat imports without duplicates, local edits, partial failures and reconnects before adding scheduled or bidirectional sync.

This does not need to make the whole convergence a prerequisite for landing the engine work. We could separate the reusable engine/CLI and general fixes from the overlapping UI/lifecycle pieces, then land the adapter as a focused follow-up. The important thing is to agree on the common boundary now so we do not end up maintaining two independent integration systems.

[michielbdejong]

A lot has happened since yesterday so I'm restarting this work in the feat/api-plugins branch and will PR it against feat/plugin-model so I can build on that. My new plan:

• create a fake API integration that just exposes a "pets" data source.
• see how it looks in the UI and how an LLM can for instance use this data source to store it in a table in a doc.
• separately build the integration-proxy, with the OAD & overlays ecosystem around it. This is closed-source for now, but I think we should open-source a light-weight version of it and only keep our throttling middleware proprietary. TODO: discuss with @joepio.
• create the machinery so that any API that is listed on the proxy is available to become a plugin data source

[michielbdejong]

Continued in #1387