Summary
Express a worklist — a queue of pending work items persisted across pages (and
drains) and consumed one at a time — from existing primitives plus one small
list-manipulation Value form, rather than a dedicated built-in construct.
A worklist is the pattern where a listing/batch response yields N items of work
(e.g. download URLs), each of which is itself fetched, decoded, and emitted. The
items must persist between pages so a long run resumes where it left off, and
only one item's body should be resident at a time (items can be large files).
It almost composes from primitives today
skopos already has the pieces to hold and walk a persisted queue:
- The queue lives in
state as an ordinary persistent field (written by
extract / progress), consistent with the rule that only extract and
progress define persisted state — no new persistence mechanism.
{first: {ref: state.queue}} reads the head; {count: {ref: state.queue}}
tests emptiness (reducers accept any list-shaped ref, not only events.*).
- A request gated
if: queue-non-empty fetches the head item; one item's decoded
body is resident per iteration, so memory is bounded for free.
- A request gated
if: queue-empty refills the queue from the listing and
extracts the listing's continuation token into another persistent state
field — so no separate durable-cursor mechanism is needed.
pagination + terminate_when loops until the queue is empty and the listing
is exhausted.
The one missing primitive
There is no Value form that returns a list minus its head (or a sub-slice).
first / last / count reduce a list to a scalar; concat / list /
select / object exist; but nothing pops or slices. Without it the queue can
be read but never shrunk, so the head is reprocessed forever.
Add a general list-slice Value form, e.g.:
{slice: {ref: state.queue, from: 1}} # drop the head
{slice: {ref: state.queue, from: 1, to: 5}} # bounded window
Then popping the head is just a progress write:
progress:
- { to: state.queue, from: { slice: { ref: state.queue, from: 1 } } }
A general slice is preferred over a single-purpose tail so it also covers
skip-N and bounded windows.
What this avoids
- No
worklist namespace and no new persistence rule — the queue is plain
persistent state.
- No durable-pagination-cursor mechanism — the listing's continuation token
persists via extract.
- Bounded memory falls out of consuming one item per iteration.
- A step toward expressing
fan_out-style per-item iteration compositionally
(per-item emit instead of merge), rather than growing a built-in.
Relationship to persistence cadence
The primitive loop is already at-least-once: an un-graceful stop reverts
state.queue to the last persisted snapshot and reprocesses that batch
(duplicates, no loss). Reducing the reprocess window is the orthogonal concern in
#61 (how often state is saved) — independent of this primitive.
Acceptance criteria
Summary
Express a worklist — a queue of pending work items persisted across pages (and
drains) and consumed one at a time — from existing primitives plus one small
list-manipulation Value form, rather than a dedicated built-in construct.
A worklist is the pattern where a listing/batch response yields N items of work
(e.g. download URLs), each of which is itself fetched, decoded, and emitted. The
items must persist between pages so a long run resumes where it left off, and
only one item's body should be resident at a time (items can be large files).
It almost composes from primitives today
skopos already has the pieces to hold and walk a persisted queue:
stateas an ordinary persistent field (written byextract/progress), consistent with the rule that onlyextractandprogressdefine persisted state — no new persistence mechanism.{first: {ref: state.queue}}reads the head;{count: {ref: state.queue}}tests emptiness (reducers accept any list-shaped ref, not only
events.*).if:queue-non-empty fetches the head item; one item's decodedbody is resident per iteration, so memory is bounded for free.
if:queue-empty refills the queue from the listing andextracts the listing's continuation token into another persistent statefield — so no separate durable-cursor mechanism is needed.
pagination+terminate_whenloops until the queue is empty and the listingis exhausted.
The one missing primitive
There is no Value form that returns a list minus its head (or a sub-slice).
first/last/countreduce a list to a scalar;concat/list/select/objectexist; but nothing pops or slices. Without it the queue canbe read but never shrunk, so the head is reprocessed forever.
Add a general list-slice Value form, e.g.:
Then popping the head is just a
progresswrite:A general
sliceis preferred over a single-purposetailso it also coversskip-N and bounded windows.
What this avoids
worklistnamespace and no new persistence rule — the queue is plainpersistent
state.persists via
extract.fan_out-style per-item iteration compositionally(per-item emit instead of merge), rather than growing a built-in.
Relationship to persistence cadence
The primitive loop is already at-least-once: an un-graceful stop reverts
state.queueto the last persisted snapshot and reprocesses that batch(duplicates, no loss). Reducing the reprocess window is the orthogonal concern in
#61 (how often state is saved) — independent of this primitive.
Acceptance criteria
{slice: {ref: <list>, from: N[, to: M]}},validated as list-shaped, with out-of-range indices clamped to an
empty/partial slice rather than erroring.
{first: {ref: state.queue}},{count: {ref: state.queue}}) covered by a fixture.cmd/skopos/testdata/driving a listing endpoint that fills astatequeue, a consume step that fetches the head and emits, and aprogresswrite that slices the head off — running across multiple pagesand composing with the [Feature] MIME-chain decoding (gzip / zip / CSV in the HTTP body) #55 decode chain.
docs/schema.md(Value forms) anddocs/runtime.mdupdated.