npm: imports specifier breaks using "own source code" as library

[4www]

Trying to make javascript imported code (local or library) available to all notebooks.

It seems not possible to import a javascript code (installed as npm package, or local code), that exports and defines web components (extending the HTMLElement class), from the observablehq.config.ts file.

Unexpected error: HTMLElement is not defined

Also tried patching the global namespace with jsDom, as when using this dependency in node, but same error.

// observable (try) patch for HTMLElement not defined with different properties
import jsdom from "jsdom";
const { JSDOM } = jsdom;

global.window = new JSDOM().window;
global.document = new JSDOM().window.document;
global.HTMLElement = new JSDOM().window.HTMLElement;
global.customElements = new JSDOM().window.customElements;
global.DOMParser = new JSDOM().window.DOMParser;

// Unexpected error: HTMLElement is not defined

Trying with this lib to import on this page (here as local js code in the ./src/ folder), in this PR.

Importing from a markdown notebook is possible, following the javascript docs, as it seems this code is evaluated in the browser.

The issue arises when importing a library or custom code from the framework config, as it seems this code is evaluated in node.

Alternative

Currently, importing the library in each notebook, and it works without error (as npm package or local code).

[Fil]

If you want to import code on the client (browser), it needs to live in the docs/ directory (you can't call code that lives outside, in ../src/). However when I install your PR locally and try to do this (by just adding a symlink from src to docs/src and fixing a few paths), I quickly hit a wall because one of the dependent libraries does not work in the browser. I probably misunderstand the question.

[4www]

Hello! Thanks a lot for your answer and taking the time to clone the PR! You understood the question perfectly.

Is there a specific reason that the js (custom) code has to live into docs/ (code is data is docs?), and can that be customized?

I'm looking into moving a few libraries to use observablehq framework, for "story book and docs" of the library functionalities. All these repos have the source in ./src/ and docs in ./docs/ which is practical to migrate, but moving the source into /docs/ feels strange, and requires making a change specific to observable: updating the other configuration for these projects, when distributing them as libraries with npm packages (ofc this can be done) — as the "source would be in docs".

I thought my custom code in ./src should work in the browser, as it was working previously with the vite configuration it used (on the main branch).

TypeError: The specifier “@sctlib/mwc” was a bare specifier, but was not remapped to anything. Relative module specifiers must start with “./”, “../” or “/”.

This error comes out of the notebook, from my imported custom code and their dependencies (though, this specific lib is built by me for browser only use cases)

Somehow, I've had to do this change in the package.json of my project when installing "@observablehq/framework": "^1.0.0",

"exports": {
    ".": "./dist-lib/index.js",
-    "providers": "./src/providers/index.js"
+    "./providers": "./src/providers/index.js"
  },

Assuming the error above comes from the same place in the @sctlib/mwc lib

My original idea was to put the import in observablehq.config.ts and import my custom code from there (so it is available globally to all notebooks), but this does not work as my lib is dependent on HTMLElement only available in browser context. Using JSDOM also did not seem to help on the global object (which i do in node with this lib) .

My plan is to:

• use observable framework as "docs and storybook", in local development, and built as a webpage presenting the lib (same as dev)
• use vite to build the src/index.js folder and entry point into a bundle distributed as npm package
• as a solution currently I imagine only mv -r src docs/src, and importing this from the notebooks.

[Fil]

a specific reason that the js code has to live into docs/

The published (exported) javascript code has to live under the site's root which corresponds to the docs/ root. So you can't go pick code that is "above" that.

It shouldn't be necessary to copy src : adding a symbolic link from docs/src to ../src is enough to allow pages to import from src (but then you'll still have many things to sort out with the npm modules).

Observable Framework is much simpler than vite in the sense that it's not trying to do server-side rendering (yet) and client-side rehydration. However, you probably don't need vite to bundle the code—you could use esbuild directly.

[4www]

Thanks again for your time and consideration to my questions.

Really cool that the symlink work!

Somehow, when trying to import my custom npm package from a cdn (intro a new html file), as it is done from within a notebook, it seems to work.

With the following code, and in the package.json using ./ to prefix named exports:

<!DOCTYPE html>
<script type="module">
import api from "https://cdn.jsdelivr.net/npm/@sctlib/mwc/api/+esm"
console.log(api)
</script>

using npx serve in the current folder where index.html is the content above

import api from "npm:@sctlib/mwc/api"
view(api)
// TypeError: error loading dynamically imported module: https://cdn.jsdelivr.net/npm/@sctlib/mwc/api

from within a notebook with local framework server

I will keep investigating, as it also seems observablehq framework complained about the ./ named exports in the original project I try to import the dependencies from (and removing the ./ stopped the error, but also maybe the export to work).

Maybe I am forgetting something in my package's export mechanisms, always getting confused.

I recognize the differences between vite and framework, and really interested in framework for all its capabilities helping to "present and explain data and code", and reason about them. Also working on a notebook/ssg (to bring org-mode notebooks to the npm/js/ssg ecosystem; where content is used instead of docs and assets is used for data/css/js/static files).

Thanks for the esbuild recommendation, I will look into it!

[4www]

Somehow it seems that my code in src/ (which is now symlinked to docs/src/) cannot itself depend on npm packages installed via npm i <package>.

It starts working if all the dependencies imports are renamed with the npm: prefix:

// from
import mwc from "@sctlib/mwc";
import initSqlJs from "sql.js";

// to
import mwc from "npm:@sctlib/mwc";
import initSqlJs from "npm:sql.js";

I am eventually using the framework wrong, as "story book" for components is not the exact use case.

It feels that if I update my library's code to fit observablehq framework, it will get complicated to both export a site with the framework and a lib with "just the custom code" (without the framework site and run time, but in my case also d3/plot/leaftlet/dompurify/mwc).

Also this lib used to work when imported, but somehow even with the npm: prefix it outputs an error

import initSqlJs from "npm:sql.js";
// SyntaxError: import not found: default

The npm: prefix also does not seem to be suported by esbuild for example when running the command npx esbuild src/index.js --bundle --outfile=dist/lib/out.js (maybe that is a configuration).

[mbostock]

Our plan is to support both CDN imports using the npm: protocol (think of this as an install that happens implicitly, invisibly in the background, whenever you import a library) and local imports from node_modules using bare module specifiers. Please upvote #360 if you are interested in the latter. So npm:sql.js would install implicitly from a CDN, whereas sql.js would need to be installed explicitly into node_modules. Our intent is to leverage the convenience of implicit install via npm: in the common case, but still give people the ability to install into node_modules using a package manager if preferred.

And since Node supports custom loaders, it will be possible to add npm: protocol support to Node — we just haven’t written it yet. #288 (Though it’s probably not quite as safe as using a package manager since you’ll typically want a lockfile to ensure that packages don’t change. Perhaps we’ll want a lockfile for Framework’s npm: protocol imports, too.)

Also, very soon we will self-host npm: protocol imports by downloading all of the imported modules; see #20 for the issue and #843 for the implementation, which is about 85% done.

[4www]

To migrate a vite project (exported as site and library) to observablehq framework:

• symlink ./src to ./docs/src (or other source path)
• rename all imports to npm:<package>
• rename all imports in docs markdown files to ../src/index.js (or match import from custom source code)
  to reach ./docs/src/index.js (symlink)

The issue seems to be that the npm: prefix (like in deno) does not work in node with vite or esbuild, to export the ./src/ code as library of it own (with a different bundle than the framework website).