Viewing events is becoming slower #4350
Replies: 6 comments 17 replies
-
|
How many cameras and roughly how many events do you have per day? What is your event retention like? Is your RPi running on an SD card or USB SSD? |
Beta Was this translation helpful? Give feedback.
-
|
Hi Nick thanks for replying, just two cameras currently. I'd say around 40-50 events per a day average so quite a few. Events retaining is different on each camera, one has active_objects + 27 days (no other config) and the other the only config is 50 days with pre_capture: 20 + post_capture: 20 No SD card but a USB 3 SSD drive (Anker usb 3 reader). Thanks again |
Beta Was this translation helpful? Give feedback.
-
|
Can you open the dev tools and see which network requests are slow? |
Beta Was this translation helpful? Give feedback.
-
|
The NAS would only affect video loading, not the database (assuming you don't have the database file on the NAS, which I could totally see slowing things down) My server had these times: |
Beta Was this translation helpful? Give feedback.
-
|
Ah, clearly my loading times here are the issue that is an insane difference :D. The database on mine is local. The only thing on the NAS is /media/frigate/recording Is your video clip storage local or on a NAS? HDD or SSD? I will try to at some point disconnect the NAS but I don't want to mess around toooo much as I kinda don't want to break it again haha. My NAS is old but the drives and 1Gb network should be enough I'd have thought. |
Beta Was this translation helpful? Give feedback.
-
|
My NAS (running unraid) is my server. It runs docker. My database, snapshots, etc. are on pooled SSDs using btrfs and my recent recordings are cached on a different pool of SSDs using btrfs with the older recordings on HDDs running xfs. I just spun my drives down and tried loading an older event and it took 3 seconds to load the master.m3u8. Once the harddrive was spun up loading a different clip took 100 milliseconds. To be honest my guess is that the Pi is struggling to serve the data quick enough |
Beta Was this translation helpful? Give feedback.
-
|
Thank you Nick for your help, it shows there is clearly something wrong with my setup. I'll do more digging, I'm glad to hear that those sorts of numbers are achievable, very impressive! Yes, I'm keeping an eye out for a replacement of my Pi, although I'm sure it was loading a lot faster back when I first installed v11. I'll test local recording first I think that will make the difference. I'll report back when I get a chance. |
Beta Was this translation helpful? Give feedback.
-
|
I think that the master.m3u8 load time has to do with the VOD module scanning the metadata for every segment so it knows where it can split the files. It has to read headers from all those files on your NAS. Once they have been read once, they are loaded from a cache until expiring. It's one of the reasons I want to try and do away with the VOD module or at least find a way to give it everything it needs to avoid that fetch. The sub labels call seems super slow. My guess is that some db indexes could improve performance there. I'm guessing the events request is being partially slowed down by the activity on the other requests. Ideally you would hit those endpoints one at a time to isolate the performance of each. |
Beta Was this translation helpful? Give feedback.
-
|
I had a similar performance issue with 6 cameras and around 5 days of data. Frigate went so slow that even listing recordings worked only sometimes, mostly it was just left spinning. I/O usage on the host (Docker on LXC on PVE) went haywire. I can't totally rule out HDD issues as I'm using a few years old 1TB USB HDD for videos, but after I deleted all clips, recordings and database, it started running smoothly again. I did have to remove clips (.png and .jpgs) in two sets because just trying to rm -rf * said something about too long command (too many files?) so I thought maybe that large amount of files could be a reason for IO issues? I may have a lot more clips from last few days as I was testing zones which may have triggered the issue, but otherwise I've run this setup for weeks without hiccup. |
Beta Was this translation helpful? Give feedback.
-
|
I was checking my NAS share whilst troubleshooting another issue and it was very interesting to see how every single recording loaded instantly and allowed me to seek instantly (Frigate was still writing current recordings to it). I haven't accessed the files via the share for many, many months now. I do however understand that the methods are vastly different between loading an event or recording in Frigate and loading the files directly, as are the clip lengths. Frigate has to do a lot of clever patching and timestamp work to process the files which are faaaaaar behind my knowledge :D. I have just one 1TB drive which is reported as healthy, I'll do a cleanup of it and start fresh to see if it helps. Edit 7th March 2023, I didn't clean the drive up but left it as is, however over the last week or so it has become super fast reviewing events in Frigate again, but no changes have been made by me. It makes no sense at all! The Pi has not updated or rebooted, the NAS certainly hasn't updated as it's been EOL for ages. Still running BETA 8 0.12.0-27A31E7. Everything is running the same on the Frigate. I think one Unify AP updated that is all I can think of, that is in a different VLAN however and everything to do with the Frigate is wired. I am still getting "Unable to keep up with recording segments in cache" errors daily but every clip from either camera is now almost instant loading like it used to be. I just don't get it at all. |
Beta Was this translation helpful? Give feedback.
-
|
Right, so a bit of an update on this thread. I have changed away from the Pi 4 to a new 4x+ more powerful Intel based PC with 8GB of RAM. I have moved to onboard SATA EXT4 formatted HDD (NAS is OFF) and still, on my h264 1440p camera I am having to wait over 30 seconds for each event to load. On my H265 1080p camera it loads events in around 5 seconds. Database is on local SSD, only recordings are on the new HDD. Again the HDD used should be plenty fast enough and is designed apparently for "20+ CCTV cameras". Looking at the network tab, all I see is it waiting over 30 seconds for seg-2-v1.ts to load.
|
Beta Was this translation helpful? Give feedback.
-
|
Just to add I did fully test and format the new local SATA HDD as ext4 which is only used for recordings (DB and everything else is stored on the M2 SSD). Thanks |
Beta Was this translation helpful? Give feedback.
-
|
NGINX logs for that: |
Beta Was this translation helpful? Give feedback.
-
|
When you come back to the page and view the same recording again, does it load just as slow the second time? |
Beta Was this translation helpful? Give feedback.
-
Yeah, just as slow - I just confirmed to be sure. |
Beta Was this translation helpful? Give feedback.
-
|
Guys so sorry I have been a bit distracted and I should have tested this first, apologies. I was reading this thread #6018 and then I forgot I was accessing via my Cloudflare tunnel. I tested locally via the host's IP and it's instant. Sorry, problem resolved. I will close this off. For info we do have a 40-50mbps Internet connection. I will look into adding some local DNS entries to redirect which might help when local. Apologies for being misleading, as this was a new bookmark I setup after moving over to my new server setup. |
Beta Was this translation helpful? Give feedback.
-
|
I experienced the same problem, but it was self-inflicted. Using an Ubuntu I5, 12 gen, with Frigate in a docker-compose container on its own 4TB NVMe drive, 13 cameras, mostly 4K, two USB corals. I had commented out port 8555 in my docker container config as an experiment, and didn't seem to suffer any ill-effects, so left it commented and forgot about it. But as my stored files grew, I experienced a slowdown only on playing stored files, not noticeable on page loading, or other operations. Restoring 8555 seems to have fixed the problem, though of course I also restarted the container in the process. FWIW. |
Beta Was this translation helpful? Give feedback.
-
Hi all,
When I first used Frigate v11 (currently I am on 0.11.1-2EADA21) I was amazed by the improvements in the event loading times, in the initial loading and especially the seek loading - all of the slow painful loading issues I had with Frigate before were suddenly resolved.
Sadly it seems over the last few weeks it's been getting quite a bit slower both for the initial loading and seeking. Even for events that just happened a few seconds ago which perhaps might even be sitting in the 1GB TMPFS RAM cache?
I have restarted my Raspberry Pi, and Frigate many times. I've restarted the NAS it records to and gave that a check over - it's only used for Frigate on a network that isn't used for anything else much at all - no high bandwidth activities going on. No changes have been made on the setup apart from just the usual system package and OS updates. CPU usage is around 35% average.
I only mount /media/frigate/recordings on the NAS, everything else is local on the SSD (lots of free space). I'll run a TRIM now on the SSD and anything else I can think of on the hardware itself and I will report back if I have any luck.
Could it perhaps be the database? Is there anything I can do to perhaps perform a maintenance task?
Many thanks
Beta Was this translation helpful? Give feedback.
All reactions