[✨] Allow loaders to be refetched programmatically and configure if they should refetch when an action is run

[jweb89]

Is your feature request related to a problem?

Yes. We have a pretty complex page which requires various api calls which we use a different loader for each. Not sure if this is feasible but it would be nice to

1. Be able to opt in or opt out of the loader being rerun when an action is run. Some loaders we don't necessarily want to be rerun when any action is run. If there are many loaders the .json from the action can get pretty large and makes unnecessary api calls etc.
2. Be able to programmatically refresh a particular loader. A use case for this would be that a user performs a particular interaction on the page and we only want to rerun a single loader

Describe the solution you'd like

export const getProductData = loader$(() => {
  return {
    product: {
      name: 'Qwik City',
      price: 100,
    },
  };
}, { strategy : 'manual' });

The strategies could be 'manual' or 'automatic' or maybe something more clear. But automatic would rerun when actions are called and manual would not. Default could be 'automatic' since that's the way it currently works now.

Additionally a loader can be rerun by calling
getProductData.refresh()

I'm sure the semantics with the naming could be improved. But if these additions could be made to allow more control over the loaders that would make the qwik data fetching even more incredible

Describe alternatives you've considered

The alternative I guess is to leave it the way it is? I'm not sure if this is even feasible as I'm not super familiar with the lower level implementation of loaders and actions

Additional context

No response

[adaliszk]

I am new to Qwik and was looking for something like this to cache the loaders and not re-run them. Ideally, I prefer a similar solution to Next.js, where they parse the cache header to set this up automatically or give you a way to set up the timer on when to invalidate this cache.

Do you know if Qwik already parses the cache header for the loader$ to effectively deliver your 1st requirement?

[jweb89]

That is a fair point. However, it doesn't address the extra data that's going to be sent from the server back to the client even if the loader is cached on the server. Which is about 150KB in our case, nearly all of which doesn't need to be sent again.

Just tested the caching, doesn't seem like it's supported(the loaders and actions didn't work on stackblitz so you'll probably need to copy the code):
https://stackblitz.com/edit/qwik-starter-ejmirx?file=src%2Froutes%2Findex.tsx,package.json

[adaliszk]

I was hoping that the cache header would trickle all that way so that re-fetching from the client side would only yield a no-change response. Thank you for testing it out!

It seems this is quite a missed opportunity.

[jweb89]

I was hoping that the cache header would trickle all that way so that re-fetching from the client side would only yield a no-change response. Thank you for testing it out!

It seems this is quite a missed opportunity.

The setup of loaders and actions doesn't really work that way. Once an action is finished running a q-data.json file is sent back to the browser which contains the result of the action and a rerun of all the loaders. This is what I'm referring to in ask 1

[manucorporat]

We want to solve this problem, but remember that actions can be

submits, ie full page reloads. In this case the loaders will need to run, just like any page refresh!

[manucorporat]

We could do:

• Some extra SPA only logic, could notify which loaders to run
• Some server-side cache level for loaders

We will looking into both strategies! so please, keep the conversation going, i would love to understand better how you plan to use this APIs.

We will collect all this feedback and try to put it together!

[jweb89]

That’s a good point. I guess maybe it makes more sense to configure the “strategy” of the action instead of the loader and the strategy could either be rerun all the loaders with a full reload or only return the data from the action.

The specific problem I was hoping this would solve is handling authentication state. We are currently fetching the user in a loader and storing it in a context(so we can update the context when the user logs out, all from the client), but it would be ideal to treat the loader as the single source of truth.

The other way I could do it is to run an action to log out the user, which would rerun the user loader. This suffered from the first problem of sending a lot of data(about 150kb) and making many unnecessary requests due to the number of loaders we have.

That said, the programmatic loaders would pretty much solve this because we could log out and then refetch only that loader. But I could see use cases where it’s unnecessary for actions to cause full reloads. So adding a strategy to actions and programmatic loaders would allow the control needed to only reload what’s needed. Such as submitting a form as an action and then refetching one loader programmatically.

[tuurbo]

The ability to control when an action$ triggers some, none or all loaders would be helpful. For example, if you have a login form and the users login was successful, fire all loaders. But if the user login fails, fire none or only specific loaders.

[juanpmarin]

@manucorporat I'm using a server$ function, how can I trigger a full page reload with it?

[DustinJSilk]

This issue has become quite a bottleneck for my app. Submitting a form in a nested route causes all routeLoaders in the parent layout files to run which in turn makes the response of my actions really slow.

[wmertens]

Bundled some duplicates. Generally speaking, the desired caching behavior seems to mostly be "forever". So if a routeloader ran once, cache it and never run it again while navigating.

Maybe this can be achieved in the serviceworker, adding SSR'd routerloaders into the browser cache?

[genki]

@wmertens It's true if the routeLoader$ is only for loading static resources, but I think the life time of the cache must be same to the resource if it is dynamic.

[genki]

Since I have read about the routeLoader$ and loadClientData to fix some issues, I came up with a way to enforce refetching programmatically.

I you want enforce refetching a route loader when SPA navigate to the page the loader related with, you can put a cache buster parameter to the href of the component.
This makes change the cache key of the client data, then you can always refetch the brand new q-data.json at the SPA navigation.

[genki]

If the route loader is a pure function, the cache buster parameter can be inferred from the arguments of the function.
To append JSON.stringify(args) is a simple solution.
I think it is great if the loadClientData can treat it automatically.
Even for cases the loader is not pure function, the optimizer may be possible to extract the implicit dependencies as well as QRLs do.

[gioboa]

We moved this issue to qwik-evolution repo to create a RFC discussion for this.
Here is our Qwik RFC process thanks.

[joshuawu]

@gioboa Is the current intended behavior that when any routeAction$ or globalAction$ is triggered, it triggers a full page re-render whereupon every routeLoader$ is triggered?

[ianlet]

In our situation, we have an interactive map where users can explore and navigate it. Everytime they move in the map, the current URL is updated to keep track of where they were (similar to what Google Maps does). Unfortunately, when this occurs, all the routeLoader$ are being re-executed even though it's almost certain the data hasn't changed. Causing a lot of unnecessary network calls.

So, simply joining this thread for another use case and sharing that having more fine-grained control on caching/deciding when to re-execute a routeLoader$ would be really useful.

[wmertens]

related: we disabled caching for q-data.json and I think it was a mistake QwikDev/qwik#7536

[DustinJSilk]

The design of bundling all loaders into a single q-data means that they're all fetched with every prefetch, navigation, or action.
For example, a settings object at the root layout is fetched and bundled into every prefetch when its only needed once.

I think this can be solved by giving us control over how the q-data objects are bundled and allowing more than 1 q-data requests for each route.

Then, let the browser cache requests based on the cach-control header and also make sure we deduplicate requests if there are multiple of the same requests at the same time for different routes sharing a q-data.

I propose introducing a new routeLoader$ config option:

bundle: undefined | 'default' | 'self'

// This is a standard routeLoader$ - there are no breaking changes.
// This is the same as using bundle: 'default'
// All routeLoaders defined like this still get bundled together into the same q-data request
const useStandard = routeLoader$(() => {
  return data
})

// This routeLoader$ gets bundled into its own q-data and can have its own cache control header.
const useStatic = routeLoader$(({ cacheControl }) => {
  cacheControl({ private: true, maxAge: 1800, staleWhileRevalidate: true })

  return data
}, {
  bundle: 'self',
})

This could even be extended at some point to allow bundling a routeLoader$ with another routerLoader, but I don't see much benefit here:

// Bundled with the useStatic loader above so it will share the same cacheControl settings
const useStatic2 = routeLoader$(() => {
  return data
}, {
  bundle: useStatic,
})

[DustinJSilk]

Of course, that makes sense. I think your idea of faking it in a single request is a really smart solution

[shairez]

@DustinJSilk great proposal! thanks!

It seems we need a thorough design doc to cover all possible use cases (caching is hard).

As this proposal is bloated, would you mind opening up a new one with your proposed design (with maybe more details / use cases) so we could discuss it specifically and move it forward?

[DustinJSilk]

Yes, and after more thought I think this can be achieved with a simpler (automatic) developer experience.
I’ll write up a new proposal soon

[shairez]

Thank you Dustin! 🙏

[DustinJSilk]

I've submitted a new proposal here: #246

[wmertens]

closing in favor of #246