[Bug]: Duplicate Communities Returned in /community/list endpoint

[fishcharlie]

Requirements

• Is this a bug report? For questions or discussions use https://lemmy.ml/c/lemmy_support or the matrix chat.
• Did you check to see if this issue already exists?
• Is this only a single bug? Do not put multiple bugs in one issue.
• Do you agree to follow the rules in our Code of Conduct?
• Is this a backend issue? Use the lemmy-ui repo for UI / frontend issues.

Summary

In my Lemmy client Echo at times duplicate entries are returned in the responses. I haven't been able to figure out why sometimes it doesn't return any duplicate entries between pages, and other times it does.

I expect there to be no duplicate entries across pages.

Steps to Reproduce

1. Unknown (this issue happens randomly, maybe 20% of the time or so?). I'm unsure how to reproduce this issue consistently.
2. Make requests for each /api/v3/community/list?type_=Subscribed&page=1 page.

Technical Details

sudo docker-compose logs > lemmy_log.out

https://fishcharliepublicbucket.s3-us-west-2.amazonaws.com/Permanent/b73e23be-97c3-48bb-bdbc-bd9f8e9e88a5/lemmy_log.txt

What OS are you trying to install lemmy on?

Ubuntu 22.04.5 LTS
In Docker

Any browser console errors?

N/A. Making direct API calls within mobile app.

---

I have also attached zip files for requests where it was working, and a set of requests where it wasn't working. If you notice in the first two responses of the one where it wasn't working the Finance community (community id: 7) was duplicated across page 1 & page 2. The only difference is in the first page it was index 1, and in the second page it was index 0.

https://fishcharliepublicbucket.s3-us-west-2.amazonaws.com/Permanent/b73e23be-97c3-48bb-bdbc-bd9f8e9e88a5/Correct.zip

https://fishcharliepublicbucket.s3-us-west-2.amazonaws.com/Permanent/b73e23be-97c3-48bb-bdbc-bd9f8e9e88a5/DuplicateCommunities.zip

Version

0.19.8

Lemmy Instance URL

eventfrontier.com

[dessalines]

I can't go through all those zip files, could you just link the specific json lines. And be sure that the actor_ids are the same.

[Nutomic]

This could happen because the data changes and the community is pushed to the following page. In that case it would be fixed once we switch the endpoint to cursor pagination.

#3872

#5429

[fishcharlie]

I can't go through all those zip files, could you just link the specific json lines. And be sure that the actor_ids are the same.

@dessalines

On the left is page=1, on the right is page=2.

As you can see the 1 index for page=1 is "actor_id": "https://beehaw.org/c/finance", and 0 index for page=2 is also "actor_id": "https://beehaw.org/c/finance",.

I'm not passing in any sort parameters.

[fishcharlie]

This could happen because the data changes and the community is pushed to the following page.

@Nutomic Even data changes not caused by user action?

[Nutomic]

If you dont pass any sort parameter then it uses hot sort by default, so that could change over time. Does the problem also happen if you pass ?sort=TopAll?

[dessalines]

Or better, sort=New .

[fishcharlie]

Yeah sort=New seems good. The thing I don't quite understand still is that the default sort option seems to be sorting based on the id number property. Except when it fails and I get this duplicate issue. So I'm still not quite understanding why it would get messed up.

I will try sort=New and report back if the issue continues to happen. It might take a bit tho just due to the rareness of the issue and my distribution process.