Dynamically generating pages from files created at runtime

[Freskoko]

Hi! loving Observable Framework so far! It feels good to work with.

My problem:
I have a background service that fetches .yaml files and saves them locally into a folder . Each file for example foo.yaml should correspond to a dynamic page (/foo) rendered via a Markdown template tat is the same for all yaml files. ( [file].md )

Is it possible for Observable to watch this directory and generate pages as new files arrive?

I have explored [file].md routes, FileAttachment, and the discussion in 855 That works well if I know the filenames at build time, but in my case, files arrive dynamically while the dev server is running, and I dont want to restart/build it every time.

I’ve managed to build one dynamic page that updates its content when files change (which was cool!!), but I would like to generate new routes automatically as files are added.

I can see one solution where i have say 5 files whos names are static, and then i update their contents with new .yaml messages. However this would limit me to only 5 data files (and therefore 5 pages) at a time (if i have understood the problem correctly).

Thanks in advance for any advice.

[Fil]

works well if I know the filenames at build time

Are you using the dynamicPaths option? In my site (pangea) I'm building a list of videos that each get converted to a page, however many they are.

However when I run it in dev, it's only getting the list once. Note that Framework is meant to be used to build static sites, and the dev server is only here to help you iterate on and preview the contents and code. If you need to poll values (or a file server) live, you would need to do it in client-side javascript.

hope this helps

[Freskoko]

Hey, thanks Fil. I managed to setup dynamicPaths much like your site, with success. I tried to take it a step further by setting up a python fastapi which would serve my yaml files, and that allowed observable to grab file data dynamically at runtime (using fetch instead of a dataloader). Using npm run dev this even made it so new pages could be generated at runtime based on new .yaml files!
However when actually building (npm run build) i ran into a bunch of trouble, as a new page route can be generated at buildtime like you mentioned, but i could not create new pages and routes based on new files, that were generated after npm run build. Given what you said, this makes sense as this is not what observable it meant for.

Thanks for your fast response and cool example!