video: sync capture: resend frame if needed to avoid capture delay #485
Conversation
|
Thank you for this PR (and all of your contributions)! Are you on our discord server https://app.lizardbyte.dev/discord? We can ask for testers there. On Linux I can test X11 with software encoding only at this point but to be honest I never noticed these delays in that scenario before. |
|
I joined the discord (same username as here on github). From looking at the encoders listed in |
|
I tested https://www.testufo.com/framerates#count=1&background=stars&pps=960 on a build with these changes. It looks like the motion is affected. It jumped more than before. OS: Linux (Flatpak on Fedora 36) |
|
Thanks, I'll look into it. I'll need to reinstall Linux on my desktop first, so it will take some time. If some encoders are too sensitive to the low capture interval, we can use a conditional check for setting the delay such as If you use moonlight-qt, it might help to enable the statistics (Ctrl + Alt + S) and see if the incoming framerate is reduced from normal. |
|
@psyke83 I had an idea, maybe not a good one, but... Would it be possible to have a setting for each application that could be named "desktop mode" (or "game mode")? Then depending on the application and the setting it would stream in the "mode" more desirable for either desktop or gaming. I don't know how difficult it would be to implement on a per app basis. |
|
I actually found a proper fix to this. Will update the PR soon after I research if the issue can also fix nvenc encoding. |
psyke83
force-pushed
the
resend_frame
branch
from
b556328 to
118e973
Compare
|
I've refreshed the patches. I may have jumped the gun in saying that I found the proper fix, unfortunately. In the case of AMF, it seems that the encoder buffers some frames even though "ultralowlatency" is supposed to encode a frame immediately after a frame is submitted. Reducing the buffer from 16 to 1 eliminates the delay, but the encoder also can no longer run at 60fps. Setting buffers to 2 allows the encoder to run at full speed and reduces latency, but doesn't eliminate the delayed frame issue completely, so by forcing a callback when a timeout the span of 3 frames passes, it greatly improves the situation to the point that I can barely notice any delay on desktop content, and I would be absolutely comforable to use Sunshine as a VNC replacement. The refreshed patch will only change behaviour for |
psyke83
force-pushed
the
resend_frame
branch
from
118e973 to
01117e3
Compare
* Add encoder flag "FORCE_CALLBACK" for encoders that don't perform well with callback capture
* Set new flag for "amdenc" and "nvenc" on Windows only.
* When flag is enabled, reduce timeout delay to 2x frame interval, and
force a callback if a timeout transpires. This should mean that a callback is
guaranteed approximately every third frame.
Before change:
* amdenc has a noticeable issue where the last received image can stutter or freeze
for a long period if the capture framerate is low (such as on desktop content).
After change:
* forced callback with a 2x frame delay interval (e.g. 32ms for 60fps) ensures
that the last received frame will not be delayed for more than two frames, and
desktop usage feels much smoother.
Note: using too low of a frame delay interval will result in a forced capture rate
of the client framerate (60fps), which can make low framerate content jittery.
Using 2x frame delay alleviate this issue, allowing the encoder rate to throttle
to ~20fps minimum, this avoiding jitter and reducing encoder strain.
The AMF encoder does not seem to output frames in realtime even if the "ultralowlatency" usage profile is used. Reducing the HW buffers from 16 -> 2 helps to reduce latency and minimize delayed frames. Note that it is necessary to set "initial_pool_size" after av_hwframe_ctx_init() in order to resize the buffer without causing issues elsewhere in the ffmpeg code. Resizing "initial_pool_size" affects the following: * https://github.com/FFmpeg/FFmpeg/blob/a78f136f3fa039fd7ad664fd6e6e976f1448c68b/libavcodec/amfenc.c#L224 * https://github.com/FFmpeg/FFmpeg/blob/a78f136f3fa039fd7ad664fd6e6e976f1448c68b/libavcodec/amfenc.c#L278-L279
psyke83
force-pushed
the
resend_frame
branch
from
01117e3 to
9710d47
Compare
|
This PR (or any other kind of hack) will be unnecessary if https://github.com/LizardByte/ffmpeg-prebuilt/pull/19 is merged and Sunshine is linked against newly patched prebuilts. Note that the patch needs to be present on the master branch for it to take effect. |
|
Closing this test PR; correct proper fix is here: LizardByte/build-deps#18 |
Description
regardless of timeout status.
This resolves the issue in which the last captured frame is delayed which
is especially noticeable in low framerate content (such as desktop streaming)
on certain encoders.
Please note that I have only verified capture delay as fixed on Windows via AMF, so nvenc and Linux capture needs to be validated before merging.
Issues Fixed or Closed
Fixes #122, #387, #412
Type of Change
Checklist