Occasional video freezing and corrupted FLV recording with nginx

[alexmck]

Describe the bug
We are seeing many customers report the video freezing for a few seconds many times throughout their streams. We also see FLV videos that don't play correctly in VLC on Mac and in other applications the audio is out of sync. When the video freezes while watching the stream, you can see that the seek bar is still progressing, so that confirms that it's not buffering. I have been able to confirm this in multiple web browsers.

The vast majority of our customers stream with a resolution of 1280x720, a bitrate of 2000 kbps, and with the IFrame set to 2. We run a fleet of servers running nginx-rtmp-module that convert the RTMP stream to HLS for consumption. There is no transcoding. We also use the record directive to save the FLV to the server for processing.

We have categorically ruled out any issues with internet speed with the customers providing us with speed test results. These same customers can then change to different applications, like Larix Broadcaster, using the same settings, and the problems go away.

I have also ruled out issues with our nginx server configuration considering OBS, Larix Broadcaster, HaishinKit and others don't have these same problems.

I have spent around 6 hours testing today using the demo RootEncoder app and I have been able to reproduce the same issue. I have been testing with the RootEncoder 2.5.2 demo app, while our app is using 2.5.5. I was able to reproduce the issue using the Camera1, Camera2, and CameraX APIs.

Internally, because of the FLV recording issue with audio and video being out of sync, we use ffmpeg to transmux the file into the MP4 container which fixes the playback and audio sync issues. Interestingly, the freeze that is observed while watching the HLS stream is not present in the FLV file (or MP4 file once transmuxed).

I have also tested forwarding the RTMP stream from our nginx servers to Cloudflare. Interestingly, this also corrects the problem with the freeze, which I believe is due to Cloudflare transcoding the input in real time. This may also explain why the issue hasn't been reported, especially if most people are sending their streams to locations that transcode in real time which corrects the problem.

Lastly, I have also done some local testing using a simple nginx container with Docker that I created (https://github.com/alexmck/docker-nginx-rtmp). Using the RootEncoder demo app in this configuration, I no longer see the random video freezes every few minutes, but the output recording still has problems playing in VLC on Mac and the audio and video are out of sync when playing in a player like IINA on Mac. Streaming with OBS and HaishinKit to this local Docker setup has none of the problems mentioned.

I suspect this is either timestamp related, or the RTMP packets are somehow slightly malformed. I don't have the technical expertise to dig down to this low level to confirm my suspicions. While I wasn't able to observe the freeze in local testing, I do believe it may be related to whatever is causing the recorded FLV file to be malformed.

To Reproduce
Steps to reproduce the behavior:

1. Run my Docker container (Installation steps are also at https://github.com/alexmck/docker-nginx-rtmp)
2. Open the RootEncoder demo app, select the rotation option, and stream to the Docker container i.e. rtmp://[YOUR COMPUTER IP]/live/AnyStreamKey
3. Once you stop the stream, have a look at the FLV recording in the /recordings folder. Observe that the file won't play in VLC on Mac and/or note that the audio and video are out of sync.

Expected behavior
The audio and video of the recording should be in sync, and there should be no freezes/pauses when consuming the HLS video stream.

Smartphone (please complete the following information):
I have been able to reproduce this on both a Samsung Galaxy S21 FE and Samsung Galaxy A15. I have also had reports from a Google Pixel 8, Samsung Galaxy S25, and a Motorola moto g14. I'm sure there are others too.

Additional context
I have been tracking this issue for several weeks now, trying to put my finger on it. I think my testing in the last day suggests there is an issue with the RootEncoder library. Please let me know if you need any further clarification or help testing. I'd love to get to the bottom of this, not only for ourselves, but to the benefit of the whole community.

[pedroSG94]

Hello,

Did you test using this?

setTimestampMode(TimestampMode.BUFFER, TimestampMode.BUFFER)

add it here:
https://github.com/pedroSG94/RootEncoder/blob/master/app/src/main/java/com/pedro/streamer/rotation/CameraFragment.kt#L80

I will check it myself too and try to reproduce it

[pedroSG94]

Hello,

For now, can you test with this PR?
#1739

[alexmck]

Your PR does fix the issue with recorded FLV videos being out of sync. Thank you.

I'm still seeing the random video freezing, but I didn't test with setTimestampMode(TimestampMode.BUFFER, TimestampMode.BUFFER) explicitly called, so I will try that and report back.

[pedroSG94]

Hello,

Thank you for the report. Can you confirm if the freeze only affect to video or affect to both (audio and video)?

[alexmck]

Hi Pedro,

During the freeze, both the audio and video stop, but the scrub bar/timeline keeps ticking. I have attached a video:

CleanShot 2025-03-16 at 10.45.47-converted.mp4.zip

I have now tested both #1739 by itself, and that same fix + setTimestampMode, and just setTimestampMode. All of these tests were conducted with the RootEncoder demo app.

Here are my results:

1. Your fixed branch resolves the issues with the recorded FLV on the nginx server being out of sync. That's awesome! But it does not resolve the freeze issue which you can see in the attached video.
2. Just testing setTimestampMode by itself fixed neither issue.
3. Testing both your fix and setTimestampMode was by far the best result. It was only on my 6th test that I saw the freeze issue. All other tests I was able to observe the freeze every single time.

[pedroSG94]

Hi Pedro,

During the freeze, both the audio and video stop, but the scrub bar/timeline keeps ticking. I have attached a video:

CleanShot 2025-03-16 at 10.45.47-converted.mp4.zip

I have now tested both #1739 by itself, and that same fix + setTimestampMode, and just setTimestampMode. All of these tests were conducted with the RootEncoder demo app.

Here are my results:

1. Your fixed branch resolves the issues with the recorded FLV on the nginx server being out of sync. That's awesome! But it does not resolve the freeze issue which you can see in the attached video.

2. Just testing `setTimestampMode` by itself fixed neither issue.

3. Testing both your fix and `setTimestampMode` was by far the best result. It was only on my 6th test that I saw the freeze issue. All other tests I was able to observe the freeze every single time.

Hello,

This are greats news. At least now, the out of sync is solved and the freeze is partially solved.
Also, I can figure that the problem could be related with the timestamp in the freeze case too.

We can try the incremental ts type for rtmp:

• Instead of use GenericStream use RtmpStream
• Set incremental ts mode:

    rtmpStream.getStreamClient().forceIncrementalTs(true)

This maybe produce sync issues or similar but we can discard that the freeze is related with this case. If this solve the freezes, I can fix the problem properly.

[alexmck]

Would you like this test to use your audio sync branch and/or setTimestampMode set?

[alexmck]

Hi Pedro,

I've just done some tests with the 2.5.7 demo app using RtmpStream and incremental ts mode as directed.

Good news! No video freezes in the 6 tests that I have performed. As expected, the audio was slightly out of sync when watching the playback live.

[pedroSG94]

Hi Pedro,

I've just done some tests with the 2.5.7 demo app using RtmpStream and incremental ts mode as directed.

Good news! No video freezes in the 6 tests that I have performed. As expected, the audio was slightly out of sync when watching the playback live.

Hello,

Thank you for the report. I will try find a solution based on this.

[pedroSG94]

Meanwhile, can you test the same but using version 2.5.8 which apply the fix for audio sync?
Maybe that is enough to fix the audio and freeze using incremental ts mode.

If not, I will see if I can implement incremental ts packet in other way.

[pedroSG94]

I did a branch for this case:
#1770

[alexmck]

Thank you Pedro. I really appreciate your support and dedication to fixing this issue.

I just finished testing 2.5.8 with the incremental ts mode enabled. I did 3 tests, and all were out of sync, and in one of the tests I witnessed the video freeze.

I will test your new branch next.

[alexmck]

I did a branch for this case: #1770

This branch was no good. I am unable to watch the stream back at all with this build. That was also using incremental ts mode and RtmpStream too.

[pedroSG94]

I did other commit for it.
If you use the setTimestamp with that values that worked for you and incrementalTs in rtmp all should work fine now.

The only problem is that this way add delay to the stream because create a cache of 2s (I think we can reduce it to 1s) and order packets by timestamp avoiding send packets without incremental ts.

[alexmck]

Thank you, Pedro. That sounds promising. I will test and get back to you.

[alexmck]

Preliminary test with this setup were good. I performed 6 tests, the first one was inconclusive, but the other 5 were in sync and there was no video freezing.

Oddly, the recorded FLV is out of sync with audio again in this setup.

Overall it seems like a real win. I'm going to perform another round of tests today.

[pedroSG94]

Oddly, the recorded FLV is out of sync with audio again in this setup.

Hello,

This is weird, if you was able to get a FLV sync without this branch, using the change for the audio I did before. Then it should be sync because the timestamp is exactly the same without modifications.

[alexmck]

I agree that it's weird. It was very much unexpected.

I ran 3 more tests today. None of them had any video freezing. The live videos were in sync. The recordings had 1 with audio in sync, and the other 2 were not in sync.

I still feel the fact that the video wasn't freezing was a big win.

Is there anything else you would specifically like me to test?

[pedroSG94]

Can you confirm that the audio is in sync without use incrementalTs?
Because I think that you should have exactly the same problem with incrementalTs and without it.
Anyway, I will finish the new branch with new incrementalTs and prepare it for merge

[pedroSG94]

Hello,
I did the final commit to the branch of incrementalTs. Also, I reduced the cache to only 300ms that seems enought to send timestamp incremental.
You can test it, I have plan to upload the change next days after fix other issues

[alexmck]

Thank you Pedro. I'm unavailable to test these change for a few days, but I will test again when I get the chance. Thank you again for your work on this 🙏