Fix bottlenecks in incoming activities queue#74
Conversation
|
Hey there. First of all, thanks for your contribution, this seems great (although I still need to test this and properly review it). Having said that, I have one note regarding this:
As you might have noticed from the open PRs, I have this repo in a bit of an abandoned state, as I'm already looking into other alternatives that are easier for me to manage. (To be fair, I have been in the "looking for alternatives" state for a long time... I'm also too lazy to move to something else 😅 ) A while ago someone forked the project into https://github.com/tinyBlogPub/microblog.pub, which already has some extra fixes on top. At the time, I was invited to help with it, but declined because I was already just keeping this in "maintenance mode". Either way, what I'm getting at is that you shouldn't rely on this repo as a stable upstream. You'll want to either rely on https://github.com/tinyBlogPub/microblog.pub or just keep your own fork. |
|
Indeed, it got a little quiet on tinyBlog.pub as I started studying and implementing a new ActivityPub engine. (I spent so long in the current code and felt like trying to refactor it, but gave up, as it is too convoluted and coupled.) I already have the bare bones of the S2S for serving blog posts, but it still doesn't have a web client (I just started looking into that this week). Let me know if you have any time and interest to give feedback. I will soon fork it inside the shared org. @brechtcs, It would be great to have your PR there to shake things up. Regards. |
|
So as I mentioned on tinyBlogPub#29, I've been running this patch on my server for several weeks, and I haven't had any blocked queues since. Before I had deal with this manually every two weeks or so. Personally, I'm also looking to move to some other AP implementation, so I'm not planning to stay on microblog.pub forever either. But this patch has significantly reduced the maintenance needed to keep my current server running while I'm looking for alternatives. So if either of you is in the same boat, merging this might help. If not, feel free to just close the PR and move on. |
|
Thank you for the PR. |
Until recently I was running a highly customised and experimental fork of microblog.pub, but I got tired of it and decided to get back to the more stable original version. I ended up using your fork because it's more up to date in its maintenance.
From my fork I did recover this significant performance improvement, which you might find useful too.
Using py-spy I found that the biggest bottleneck was updating reply counters whenever a Delete activity came in (of which there are lots). This improves drastically when
inReplyTois stored in a dedicated column, rather than relying on SQLite'sjson_extractfor this.I also added two missing indices on the
incoming_activitytable.