[PLAN] Download folder as text

[hoyla]

Download Folder as Text

Summary

Add an admin-only "Download folder as text" action to the workspace-tree context menu, scoped to a selected folder and its descendants. It reuses @twrichards's existing "Download workspace as text" machinery and the folder-search descendant logic, so it's mostly a wiring exercise. Counts and downloaded contents reconcile with the existing workspace figure because they use the identical counting method.

Stage 1 (below) is self-contained and shippable with no behaviour change to the existing workspace download.

Attachment/archive sub-documents and server-side URI derivation are deliberately deferred to Stage 2, which should ride the lazy-loading migration (at which point the Download workspace as text" feature will also need adjustment).

Background

Three existing pieces give us most of the ingredients for this feature:

• "Download all text" (see add new (admin only) option to download all the text of a workspace #691) is almost entirely frontend. DownloadTextModal walks the frontend tree to collect {blobUri → path}, batches the URIs to POST /api/workspaces/:id/text (getTextForBlobs), wraps each in ----- START/END OF … ----- markers, and splits the output into numbered .txt files. The text endpoint takes an arbitrary blob-URI list scoped by workspaceId, so it is already folder-agnostic.
• "Search in folder" (see Giant search in folder: use targeted Cypher for blob rather than fetching the entire workspace #740) scopes work to a folder subtree. Its reusable nugget is getBlobUrisInWorkspaceFolder(user, workspaceId, folderId) in Neo4jAnnotations.scala, which returns all descendant file URIs via the (:file)-[:PARENT*0..]->(folder) traversal. (Needed for Stage 2, not Stage 1.)
• The workspace-tree context menu already distinguishes folder vs file (isWorkspaceNode / isWorkspaceLeaf) and already gates admin-only items (e.g. "Reprocess folder contents").

Why two stages

The only consistency risk is the word count, which is shown to the user ("This workspace has a total of X words…"). Divergence has two independent sources:

1. Counting method — solved by reusing the existing Elasticsearch Painless aggregation unchanged.
2. Membership — three different notions of "what's in scope": the ES workspace tag, the Neo4j tree descendants, and parentBlobs sub-documents (attachments, archive contents) that aren't tree nodes. Folder-search deliberately includes parentBlobs; the current workspace download does not.

Including parentBlobs would make folder sums exceed the current workspace total and would require changing the text-fetch authorisation model (sub-documents generally aren't workspace-tagged, so the current security filter returns empty for them). That's real work and is bundled into Stage 2, where it naturally pairs with the lazy-loading migration (both require server-side URI derivation). Stage 1 stays on tree nodes only, where the existing endpoints work untouched.

Stage 1 — Folder download (tree nodes only)

Backend

The only gap is a folder word count (ES has no folder tag). Mirror the existing getText endpoint, swapping the workspace-tag filter for an id-list filter — same aggregation, so the method matches the workspace count.

1. Route (backend/conf/routes), beside the existing :workspaceId/text:

   POST  /api/workspaces/:workspaceId/wordCount  controllers.api.Workspaces.getWordCount(workspaceId: String)
   

2. Controller (controllers/api/Workspaces.scala) — model on getText: check access via annotation.getWorkspaceMetadata, validate body as List[String], call the new index method.
3. Index (services/index/Index.scala + ElasticsearchResources.scala) — add getWordCountForBlobs(workspaceId, blobUris): Attempt[Long]. Reuse the getTotalWordCountForWorkspace scripted-metric aggregation verbatim, but change the query to:

   must(idsQuery(blobUris), buildWorkspaceIdFilter(workspaceId))

   Keeping the workspace filter preserves the same security property as getTextForBlobs. Use size(0) — returns only the aggregate, lighter than the text fetch.

Frontend

4. API client (services/WorkspaceApi.ts) — add getWordCountForBlobs(id, blobUris), mirroring getWorkspaceText.
5. Generalise the modal (components/workspace/DownloadTextModal.tsx):
   • Accept an optional rootEntry (default workspace.rootNode) and label (default workspace.name).
   • Build the full path map from workspace.rootNode as today, but filter its keys to the folder's descendant URIs — this preserves full workspace-root-relative paths in the file headers (provenance) while scoping contents to the folder.
   • When rootEntry is supplied, source the count from getWordCountForBlobs(workspace.id, folderBlobUris) instead of getWorkspaceTotalWordCount. The workspace path is untouched.
   • Adjust displayed copy to say "folder" vs "workspace".
6. Context-menu item (components/workspace/Workspaces.tsx, renderContextMenu) — add an item gated on isAdmin && isWorkspaceFolder && !isRemoteIngest (same gate as "Reprocess folder contents"). On click, open a DownloadTextModal instance with rootEntry/label set to the selected folder, following the existing reprocessFolderModalEntry modal-state pattern.

Consistency & security notes

• Reconciliation: workspace count uses the tag filter, folder counts use the id-list filter — same Painless script, so the sum of top-level folders ≈ workspace total, modulo any tag↔tree drift. Attachments (the real reconciliation-breaker) are out of scope here.
• No auth-model change: tree files are workspace-tagged, so the existing workspace filters Just Work. No parentBlobs authorisation needed yet.
• Lazy-loading: Stage 1 rests on the same "full tree in frontend memory" assumption the existing workspace download already relies on — no new risk introduced. Stage 2 is the shared migration point.

Acceptance criteria

• Admins see "Download folder as text" on right-clicking a folder; non-admins and file selections do not*.
• The downloaded file set matches the folder's tree descendants.
• The pre-download word count is shown for the folder and uses the same method as the workspace figure.
• The sum of top-level folder counts ≈ the workspace total (allowing for tag↔tree drift).
• No change to the existing workspace "Download all text" behaviour or figure.
• Backend ITest for getWordCountForBlobs (runs in CI; ITests don't run locally).

Stage 2 — Sub-documents + server-side derivation (deferred; rides lazy-loading)

Bundled because both need server-side URI derivation:

• Move folder and workspace URI derivation server-side via getBlobUrisInWorkspaceFolder — fixes lazy-loading fragility for both.
• Include parentBlobs sub-documents (attachments, zip/archive contents) using folder-search's buildWorkspaceBlobFilter authorisation model, so sub-blobs are authorised by parent membership.
• Re-point the workspace count/download to the same derivation so reconciliation holds with attachments included.
• Behaviour-change heads-up for that PR: the existing "Download all text" will start including attachment/archive text it currently omits, and its word count will rise.

Open questions

• Full workspace-root-relative paths in the file headers (recommended, for provenance) vs folder-relative paths?
• Filename convention for folder downloads — {folder name} ({workspace id}).{n}.txt or include the workspace name?

• We could easily add "download file as text" later if so desired.