Skip to content

[Feature] List-slice Value form ({slice: {ref, from, to}}) — express worklists from primitives #60

Description

@P1llus

Summary

Express a worklist — a queue of pending work items persisted across pages (and
drains) and consumed one at a time — from existing primitives plus one small
list-manipulation Value form, rather than a dedicated built-in construct.

A worklist is the pattern where a listing/batch response yields N items of work
(e.g. download URLs), each of which is itself fetched, decoded, and emitted. The
items must persist between pages so a long run resumes where it left off, and
only one item's body should be resident at a time (items can be large files).

It almost composes from primitives today

skopos already has the pieces to hold and walk a persisted queue:

  • The queue lives in state as an ordinary persistent field (written by
    extract / progress), consistent with the rule that only extract and
    progress define persisted state — no new persistence mechanism.
  • {first: {ref: state.queue}} reads the head; {count: {ref: state.queue}}
    tests emptiness (reducers accept any list-shaped ref, not only events.*).
  • A request gated if: queue-non-empty fetches the head item; one item's decoded
    body is resident per iteration, so memory is bounded for free.
  • A request gated if: queue-empty refills the queue from the listing and
    extracts the listing's continuation token into another persistent state
    field — so no separate durable-cursor mechanism is needed.
  • pagination + terminate_when loops until the queue is empty and the listing
    is exhausted.

The one missing primitive

There is no Value form that returns a list minus its head (or a sub-slice).
first / last / count reduce a list to a scalar; concat / list /
select / object exist; but nothing pops or slices. Without it the queue can
be read but never shrunk, so the head is reprocessed forever.

Add a general list-slice Value form, e.g.:

{slice: {ref: state.queue, from: 1}}          # drop the head
{slice: {ref: state.queue, from: 1, to: 5}}   # bounded window

Then popping the head is just a progress write:

progress:
  - { to: state.queue, from: { slice: { ref: state.queue, from: 1 } } }

A general slice is preferred over a single-purpose tail so it also covers
skip-N and bounded windows.

What this avoids

  • No worklist namespace and no new persistence rule — the queue is plain
    persistent state.
  • No durable-pagination-cursor mechanism — the listing's continuation token
    persists via extract.
  • Bounded memory falls out of consuming one item per iteration.
  • A step toward expressing fan_out-style per-item iteration compositionally
    (per-item emit instead of merge), rather than growing a built-in.

Relationship to persistence cadence

The primitive loop is already at-least-once: an un-graceful stop reverts
state.queue to the last persisted snapshot and reprocesses that batch
(duplicates, no loss). Reducing the reprocess window is the orthogonal concern in
#61 (how often state is saved) — independent of this primitive.

Acceptance criteria

  • A list-slice Value form {slice: {ref: <list>, from: N[, to: M]}},
    validated as list-shaped, with out-of-range indices clamped to an
    empty/partial slice rather than erroring.
  • Reducers over a state-list ref ({first: {ref: state.queue}},
    {count: {ref: state.queue}}) covered by a fixture.
  • Golden under cmd/skopos/testdata/ driving a listing endpoint that fills a
    state queue, a consume step that fetches the head and emits, and a
    progress write that slices the head off — running across multiple pages
    and composing with the [Feature] MIME-chain decoding (gzip / zip / CSV in the HTTP body) #55 decode chain.
  • docs/schema.md (Value forms) and docs/runtime.md updated.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions