Skip to content

Is it really normal scraper is slow at processing records? #388

@benoit74

Description

@benoit74

For https://farm.openzim.org/pipeline/cd36db85-6748-4510-a3b6-b4a94425fa0a/debug, it took 80.43 hours (from 2025-12-19 15:17:10 to 2025-12-22 23:42:44) to process 24.20 millions records in the Questions_Meta phase, or in average 5014 records per minute, 83 records per second.

This looks amazingly slow to me, especially since this Questions_Meta phase is not fully processing the record, only preprocessing it and storing important metadata in Redis. I'm wondering if there is something wrong somewhere, at least it seems extraordinary slow to me.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions