8.1.0 prerelease used 59gb of memory

[balupton]

After upgrading to the 8.1.0 prerelease from 8.0.7 on a m1 Mac mini running the latest macOS resulted in the force quit out of memory macOS modal after about an hour of use, in which tribler was the culprit at 59gb of memory on a 16gb of ram machine. I had to force quit it from that modal as the menu bar icon was unresponsive.

I have I guess 200 torrents in which half of them are completed.

Haven't reopened it since the force quit.

Let me know what steps are needed to send the logs.

[teddyrogers]

I have noticed this on Windows too. Last time I check Tribler was using 5.8GB.

[absolutep]

After upgrading to the 8.1.0 prerelease from 8.0.7 on a m1 Mac mini running the latest macOS resulted in the force quit out of memory macOS modal after about an hour of use, in which tribler was the culprit at 59gb of memory on a 16gb of ram machine. I had to force quit it from that modal as the menu bar icon was unresponsive.

I have I guess 200 torrents in which half of them are completed.

Haven't reopened it since the force quit.

Let me know what steps are needed to send the logs.

Did you change Hops?

This happened with me [sudden rise in size of Tribler folder] also.

I changed Hops of my torrents and then suddenly my Tribler folder was equivalent to 80 GB.

All these torrents of mine, were already present in BiglyBT, so when I opened BiglyBT - all those torrents suffered disk error & automatically deleted themselves.

I have put them on redownload but some torrents were very obscure - hopefully I can get them back.

I think this error is related to this = #8483

But who knows

[teddyrogers]

I think my issue may have been related to the debug logs. When the logs are open it is consuming excessive amounts of memory in the browser, not specific to the Tribler process. I have seen it use up to over 8GB of memory.

[balupton]

No change in hops, they have all been 3 hops.

I did not have the web client open at the time.

[teddyrogers]

I checked Tribler process today and it is reporting 7.2GB....

Unfortunately I restarted Tribler before checking where the memory is being used.

[mouse-the]

Since upgrading to 8.1.0 I am also seeing uncontrolled memory usage.
Debian Linux, 24g ram, 1g swap. Github .deb 8.1.0
I do have a 3 digit number of downloads, all but ~10 completed.
This morning I managed to observe something that probably also happened twice yesterday.
In short, after running unattended for 6 hours, tribler had used all the memory.
kswapd was running the CPU hard,
tribler had more than half the ram and 60g of virt
I think the KDE lockscreen did not manage to come up until tribler shut down.

I have tribler's output redirected to /dev/null to prevent it from filling my root partition, so I have not been able to look for hints there.

Based on casually observing top while writing this, tribler seems to be steadily consuming memory, maybe on the order of a gig per hour. ...and already 42g of virt...

Oh, poking around in the journal confirms the kernel killed tribler:

Mar 09 03:03:21 mouse23 systemd[1139]: Started app-org.tribler.Tribler-4cd2900cefb443c68466c7f5074053ed.scope - Tribler - P2P /Bittorrent/Youtube client.
Mar 09 11:37:19 mouse23 kernel: [ 25337] 1000 25337 15914039 3590121 35635200 5895 200 tribler
Mar 09 11:37:19 mouse23 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/[email protected]/app.slice/app-org.tribler.Tribler-4cd2900cefb443c68466c7f5074053ed.scope,task=tribler,pid=25337,uid=1000
Mar 09 11:37:19 mouse23 kernel: Out of memory: Killed process 25337 (tribler) total-vm:63656156kB, anon-rss:14360476kB, file-rss:0kB, shmem-rss:8kB, UID:1000 pgtables:34800kB oom_score_adj:200
Mar 09 11:37:18 mouse23 systemd[1139]: app-org.tribler.Tribler-4cd2900cefb443c68466c7f5074053ed.scope: A process of this unit has been killed by the OOM killer.
Mar 09 11:37:20 mouse23 kernel: oom_reaper: reaped process 25337 (tribler), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Mar 09 11:37:20 mouse23 systemd[1139]: app-org.tribler.Tribler-4cd2900cefb443c68466c7f5074053ed.scope: Failed with result 'oom-kill'.
Mar 09 11:37:20 mouse23 systemd[1139]: app-org.tribler.Tribler-4cd2900cefb443c68466c7f5074053ed.scope: Consumed 4h 32min 11.684s CPU time.

...and another glance at top suggests that the memory has leveled off for now...
Let me know if I can help.

Update from 2 hours later; the memory use has gone down a bit, so it is not a simple case of memory use steadily increasing...

[teddyrogers]

Logged on today, checked Task Manager and Tribler was using 11.5GB memory. Took a screenshot...

...then went to check process memory usage to diagnose and it dropped back down to 1.2GB???

[qstokkink]

Thanks for the input everyone. I think we hit this bug (inside our libtorrent dependency, which has been updated to support Bittorrent v2): arvidn/libtorrent#6667

If I read that (massive) issue correctly, the default disk writing mode of mmap is causing these issues. We should be able to fix this in Tribler by setting the default disk writing mode (disk_write_mode) to mmap_write_mode_t.always_pwrite. We'll try that for the next pre-release.

[teddyrogers]

@qstokkink I installed Tribler 8.0.7 and ran some tests in comparison to 8.1.0 and it looks like the current version is mapping the torrent data files to memory. I don't see the mapping occurring in Tribler 8.0.7. I think this also explains why I was seeing memory usage spiking and then dropping off...

[HanabishiRecca]

disk_write_mode only affects writes. Seeding would still hog the memory.
In qBittorrent we use this workaround: qbittorrent/qBittorrent#21300

[qstokkink]

@HanabishiRecca thanks for the help! Setting the mmap_file_size_cutoff to the max int was not quite obvious to me from the docs. I imagine that took quite a bit of effort to find.

Re-opening this issue until the above is implemented.

[teddyrogers]

Tribler 8.1.1, there is a large amount of memory pages being allocated for torrent data files. I believe this to be a continuation of #8485.

A current Tribler session is using 8.95GB of memory...

...here are the commits to Triblers memory map...

[teddyrogers]

@qstokkink Tribler 8.1.2, whilst not as savage as previous 8.1.x releases memory usage is still a problem.

Do you require further information to resolve?

[balupton]

just stating in advance that I (the OP) won't be available to test or feedback anything until end of april, so feel fee to close or reopen if it is a peovlem

[qstokkink]

Unless @HanabishiRecca has any more magic settings for us to play with, I think this is just the cost of supporting v2 torrents. So, regarding:

Do you require further information to resolve?

I think that - at this point - we cannot resolve this and the only way we can further reduce memory, is by going back to BitTorrent v1 again. This is probably also the reason why qBittorrent offers a v1 and v2 option.

[HanabishiRecca]

mmap_file_size_cutoff mitigates the mmap problem. E.g. my server instance (qbittorrent-nox) never goes beyond even 100M.

So I see only 2 reasons:

1. The setting is not applied somehow.
2. There is another unrelated memory leak.

[qstokkink]

That is useful input! Tribler didn't have this memory consumption before: if it is possible to not go over 100M, we must still have a chance to reduce memory consumption. In turn, if there is still a chance to reduce memory usage, I'll keep this issue open.

[HanabishiRecca]

I think this is just the cost of supporting v2 torrents.

Btw about that. Libtorrent 1.x is effectively unmaintained and at this point doesn't even compile against the last version of Boost (arvidn/libtorrent#7796). So it's only matter of time when it would became completely unusable.

[HanabishiRecca]

I also noticed that Tribler uses an outdated version of libtorrent (2.0.9). It is highly advised to upgrade to the latest (2.0.11 atm) as we also pushed a lot of fixes on its side in conjunction.

[mouse-the]

I let 8.1.2 run last night, and it may have been a small improvement over 8.1.0: 10.5 hours instead of 8.5, but that might be because it got 17g of ram instead of 14g.
There was a big reduction in total vm size: 20g instead of 64g.

Mar 16 19:57:59 mouse23 systemd[1154]: Started app-org.tribler.Tribler-a744a0e41eae4d9b8e66d3748733bce0.scope - Tribler - P2P /Bittorrent/Youtube client.
Mar 17 06:21:04 mouse23 kernel: [ 40462] 1000 40462 5212866 4461149 37769216 147908 200 tribler
Mar 17 06:21:04 mouse23 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/[email protected]/app.slice/app-org.tribler.Tribler-a744a0e41eae4d9b8e66d3748733bce0.scope,task=tribler,pid=40462,uid=1000
Mar 17 06:21:04 mouse23 kernel: Out of memory: Killed process 40462 (tribler) total-vm:20851464kB, anon-rss:17844588kB, file-rss:0kB, shmem-rss:8kB, UID:1000 pgtables:36884kB oom_score_adj:200