Set the depth of the framebuffer queue?

[ambrusc]

I'm debugging some pretty significant input delay which seems on the order of ~100 ms or more. This may not sound like a lot, but it's very noticeable if you compare the scroll responsiveness of even something like the web browser.

My app is using the signal from renderer->beginFrame(...) to schedule frames and it appears as if several (8-ish?) frames are drawn exremely quickly, then the frame time settles to the screen refresh of 60hz. I've tried this on Macos (intel, Metal) and Manjaro Linux (Vulkan & OpenGL, but I can't remember which I used last) with similar behavior: lots of input delay.

The scrolling window is using ImGui and I'm also wondering if there's additional delay after calling setBuffer[At](...) before those results are actually rendered. How many frames does it usually take to sync vertex buffers to the GPU on desktop?

So my two questions:

1. Is there a way to set the depth of the framebuffer queue used by Filament? Ideally I'd like it to be 2-ish.
2. How many frames can it take for setBuffer to actually propagate to the GPU? What is rendered prior to that data being available on the GPU?

[ambrusc]

If is slow my mainloop down and force it to call beginFrame at most once every 1/60th of a second, the input delay is improved significantly, but still worse than a web browser.

[romainguy]

Looks like you are stuffing the display queue, which will cause those delays. You should instead schedule your frame using the platform's vsync signal.

[ambrusc]

Thanks for the very fast response!

That's what I was afraid of, but I don't see an obvious way to do this.

I'm using GLFW for cross platform window/input. Filament controls the buffer queue, so it's not obvious to me if it even makes sense to call glfwSwapInterval or glfwSwapBuffers at all, especially when using the vk or metal backends of Filament.

The other options I see are:

• SwapChain::setFrameScheduledCallback (only for metal, so won't work on linux)
• SwapChain::setFrameCompletedCallback (only for metal, so won't work on linux)
• Engine::flushAndWait which blocks the calling thread and isn't recommended per comments

If I could set the buffer queue depth, this wouldn't be a huge problem with the current filament APIs. Alternatively, if I knew how many buffers were currently scheduled but not yet presented, I could throttle as well.

Recommendations?

[romainguy]

@pixelflinger

[pixelflinger]

We don't have a good way to do what you want. The way it should work is that you render a frame at h/w vsync speed and then we'll take care of making sure we don't let you enqueue another one if the GPU isn't done with the previous one.

If you can't schedule your frames every vsync, this is not goin to work.

There might be one solution though but it's a little involved -- You could try to subclass the corresponding Platform class, which will let you know when we're calling "swap buffers" (the hook is called commit), with that, you might be able to build what you want, i.e. controlling how many buffers are in flight.

The latency is never going to be good though if you can't synchronize with the h/w vsync.

One dumb way to do this is to spawn a thread that just loop on a "swapbuffers" (you'll need a separate context for that), each time it returns, that basically gives you a "vsync signal", you could for instance release a condition variable, which your main thread waits on.

[ambrusc]

I really appreciate the additional details.

The way it should work is that you render a frame at h/w vsync speed and then we'll take care of making sure we don't let you enqueue another one if the GPU isn't done with the previous one.

I think my attempt with slowing down the frame scheduling to vsync speed (i.e. never exceed the refresh rate) if not the actual vsync signal is close to doing this. However, I can't find any API that prevents the scheduling of additional frames before the GPU is ready. Are you saying this is the API that doesn't currently exist but should? Or am I misunderstanding something :-)

The latency is never going to be good though if you can't synchronize with the h/w vsync.

I think the problem is more subtle -- would love to hear your feedback. Even if I were to schedule each frame at vsync signal, there's no guarantee that the Filament/render thread is active, or that the GPU would actually finish rendering in time for the next presentation. If it doesn't, my mainloop could happily schedule another frame after the the next vsync, even though the prior one isn't done because it doesn't actually know when Filament has called "present." Every time this happens, another frame is added to the buffer queue, increasing latency, and my mainloop has no way to know this.

In practice, with my slowed mainloop, I notice the latency creeping up every time I bring another window to the foreground or do something intensive on the system. My guess is that the subtle problem I just described is the cause, but again, I could be way off.

One dumb way to do this is to spawn a thread that just loop on a "swapbuffers"

I suspect this would have the same issue I just described? (I.e. I don't actually know when Filament has called "present".)

try to subclass the corresponding Platform class

This looks promising, but it appears only the OpenGLPlatform has commit (on Linux I'm using Vulkan and on Macos I'm using Metal, though I guess I could switch to the Opengl backend for both).

#6675 says "the sample schedules frames badly using a fixed timestep of ~16ms, it doesn't try to use vsync at all." #2014 asking about vsync callbacks has a link to #636 but that issue is also closed. Was there functionality added, or it was just an old issue that got de-prioritized?

How is this solved on other platforms? Is there a core issue where Filament doesn't have enough information to provide a good API, or is it that it just hasn't been implemented yet?

Apologies for all the questions. I've been reading all the issues, trying a bunch of different experiments, and reading the filament platform code to try and make heads or tails of this for several days and I've gotten as far as I can without asking for help :-)