New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

关于去重性能 #10

Open

6api opened this issue Aug 3, 2017 · 0 comments

6api commented Aug 3, 2017

看了源代码中去重使用的2种方法，一个是md5直接放redis set，这个数据量到百万千万后性能不行
另一个使用bloomfilter映射redis bitmap，这2者在爬取URL数量在千万级性能差距有多少？

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment