feat(memory-storage): implement listRequests method#3727
feat(memory-storage): implement listRequests method#3727harryautomazione wants to merge 2 commits into
listRequests method#3727Conversation
There was a problem hiding this comment.
Pull request overview
Adds missing RequestQueueClient.listRequests() support to local @crawlee/memory-storage to reduce the API gap versus the platform client and enable programmatic inspection/pagination of request queues in local environments.
Changes:
- Extended
@crawlee/typesRequestQueueClientwithlistRequests()plus its options/result interfaces. - Implemented
listRequests()in the memory-storageRequestQueueClientusingMapiteration withlimit+exclusiveStartId. - Added unit tests covering default listing,
limit,exclusiveStartId, and basic pagination.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| packages/types/src/storages.ts | Adds listRequests() typing and associated option/result interfaces to RequestQueueClient. |
| packages/memory-storage/src/resource-clients/request-queue.ts | Implements the new listRequests() method in the local memory-storage request queue client. |
| packages/memory-storage/test/request-queue/list-requests.test.ts | Adds tests validating listing behavior and pagination semantics. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| export interface RequestQueueClientListRequestsOptions { | ||
| limit?: number; | ||
| exclusiveStartId?: string; | ||
| } |
| export interface RequestQueueClientListRequestsResult { | ||
| limit: number; | ||
| exclusiveStartId?: string; | ||
| items: RequestSchema[]; | ||
| } |
| const queue = await this.getQueue(); | ||
| const items: storage.RequestSchema[] = []; | ||
|
|
||
| let isAfterExclusiveStartId = !exclusiveStartId; | ||
|
|
||
| for (const [requestId, storageEntry] of queue.requests.entries()) { | ||
| if (!isAfterExclusiveStartId) { | ||
| if (requestId === exclusiveStartId) { | ||
| isAfterExclusiveStartId = true; | ||
| } | ||
| continue; | ||
| } | ||
|
|
||
| if (items.length >= limit) { | ||
| break; | ||
| } | ||
|
|
||
| const request = await storageEntry.get(); | ||
| items.push(this._jsonToRequest<storage.RequestSchema>(request.json)!); | ||
| } |
barjin
left a comment
barjin
left a comment
There was a problem hiding this comment.
Thank you @harryautomazione ,
the changes look all right to me, although I agree with the Copilot comments.
If you want to proceed with this PR, please go over the Copilot comments and address these issues. Cheers!
|
Hi @barjin! Thank you for the review. |
Description
Closes #2781
The method
RequestQueueClient.listRequestsexists for platform usage (inapify-client), but it was entirely missing from the localMemoryStorageclient (@crawlee/memory-storage). This discrepancy prevented users from inspecting request queues programmatically in local environments.This PR implements the missing method to ensure full API parity:
@crawlee/types: ExtendedRequestQueueClientto includelistRequestsalongside its options/results interfaces (RequestQueueClientListRequestsOptions, etc.).@crawlee/memory-storage: Implemented the core logic using the Map insertion order. It properly handles both thelimitandexclusiveStartIdparameters to allow safe, deterministic pagination of local requests.list-requests.test.tsto simulate and verify single, limited, and fully paginated extractions.Type of change
Checklist: