Is Framework appropriate for refreshable data? What about live?

[ddxv]

Hello, I'm totally new to framework and wanted to learn more about it, can the data be somewhat live? I ask because the home page describes it as:

"Data loaders generate files — typically static snapshots of data — at build time."

But wouldn't this mean the site is somewhat locked in place at the time of build? Can the data be refreshed at set intervals like once a minute or is that just not in spirit with the project?

[Fil]

A page can totally consume live data the usual way—directly from the browser, using fetch or d3.json for example. You don't have to use data loaders, they're just an added possibility.

[ddxv]

Actually, something I should verify my understanding, when you say "at build time" my assumption was that meant when the JS app is built and deployed for production, but perhaps that's not what that means?

For example, if you 'build' the data based on Feb 20th. And a couple weeks go by without a new deploy, would the data still show Feb 20th?

I occasionally want to add at build time or cache data that I think changes very rarely so that it is easy to use, but usually I put these in materialized views or data aggregation pipelines.

[Fil]

if you 'build' the data based on Feb 20th. And a couple weeks go by without a new deploy, would the data still show Feb 20th?

If we're speaking of the data snapshots created with data loaders: yes. They are static files.

[mbostock]

And you can use continuous deployment to refresh the data as often as you like.

[tx2o]

but in some projects the raw live data is pretty big don't want end users' browsers to ingest that raw live data, something like a spreadsheet shared by an upstream project, it might have ~50k rows x 200 columns

https://docs.google.com/spreadsheets/d/${id}/export?format=${format}&gid=${gid}

this raw export live url might be too expensive to load in browsers, so I use a framework data loader to extract only some rows x 20 columns and might do some filtering out outlier values on the fly, and let my end users to load only a small chunk of clean data from raw upstream live url?

so is to rebuild and redeploy at every day every hour the only way?

and can request this feature?

Can the data loader be refreshed at set intervals like once a minute or is that just not in spirit with the project?

some thing like a POST admin flag to kick a particular data loader be refreshed manually? instead of redeploy the whole app?

[Fil]

Can the data loader be refreshed at set intervals like once a minute

The way Framework operates is by building a whole static site. So the basic answer could be to run CI every minute. However this may not be practical, because building a whole site may take more than a minute, and anyway it's only one file that needs to be updated.

So, I'd suggest to create a smaller project that just runs the same script as your data loader and updates a unique file, that you can then query from framework as a "remote" resource. That script could run in CI every minute, but it might be more ecological to only run it on-demand as an endpoint in val.town, or a “serverless” function, with proper caching.