Use this guide when local Engram saves work but cloud sync does not advance. The safest rule is simple: local SQLite is the source of truth; cloud is replication. Back up the local database before repairing anything.
Run these commands first:
engram version
engram cloud status
engram cloud upgrade doctor --project <project>
engram sync --cloud --status --project <project>Check three things:
| Signal | Healthy value | What it means |
|---|---|---|
engram version |
Latest release on client and server | Client/server version skew can block old chunk sync |
cloud status |
configured + auth ready | CLI has a server URL and runtime token |
doctor |
ready or actionable repair |
Local metadata is safe to upload |
last_acked_seq |
Advances after sync | Cloud accepted the pending journal |
If the dashboard shows 0 observations but local saves exist, the cloud server has not accepted the client's pending sync yet. Do not delete local data.
Use the supported commands exactly:
engram cloud config --server https://your-cloud-host
engram cloud status
engram cloud enroll <project>
engram sync --cloud --project <project>engram cloud config --status is not the documented status command. Use engram cloud status.
Cloud auth token is runtime config:
export ENGRAM_CLOUD_TOKEN="your-token"The local ~/.engram/cloud.json stores the server URL and may also store a token fallback. ENGRAM_CLOUD_TOKEN takes precedence over any token in cloud.json; if the env var is unset, Engram falls back to cloud.json.token. This fallback is intentional (issue #343) for use cases such as background autosync where exporting the env var on every shell is not practical.
engram cloud status --project <project>
engram cloud unenroll <project>
engram cloud config --clearcloud status reports effective server/auth configuration separately from project enrollment. Use --project to see whether that project is enrolled.
cloud unenroll is idempotent and stops future replication eligibility for that project. It does not delete local data, remote history, or pending journal rows, and it does not cancel an already in-flight push.
cloud config --clear clears only the persisted cloud.json server URL and token. Active ENGRAM_CLOUD_SERVER and ENGRAM_CLOUD_TOKEN overrides remain effective and are reported by the command and status output; unset them separately when you need them inactive.
When a project's cloud data was deleted or recreated while the correct local data is already acknowledged, replay the current local project state with:
engram cloud upgrade remirror --project <project>The project must already be enrolled and the configured cloud credentials must be authorized for it. Remirror creates new project-scoped upsert and locally represented tombstone mutations, then sends them through the normal cloud export path. It preserves existing acknowledgement and attempt history; it does not reopen acknowledged rows, delete mutation history, or reset last_acked_seq. Re-running the command is safe because remote entities use stable identities and upsert semantics.
This error means the legacy chunk upload endpoint rejected a payload because the client-provided chunk_id did not match the server-computed canonical hash.
Upgrade both sides to v1.14.8 or newer:
brew update
brew upgrade engram
engram versionRedeploy or restart the cloud server so the server binary also runs v1.14.8 or newer.
Then retry:
engram sync --cloud --project <project>
engram sync --cloud --status --project <project>In v1.14.8, the server treats the client chunk_id as advisory. The server validates and canonicalizes the payload, computes its own chunk ID, stores using the server-computed ID, and returns that ID. Valid payloads no longer get blocked by client/server canonicalization drift.
This is the common legacy manual-save blocker:
session payload directory is required and cannot be inferred from local state (seq=N entity=session op=upsert)
Another legacy blocker can appear for observation upserts:
observation payload missing required upsert fields: title (seq=N entity=observation op=upsert)
canonicalize cloud chunk: mutations[768]: observation payload title is required for upsert
You may also see the failure only during the final server push, even after doctor says the project is ready:
write chunk: cloud: push chunk ...: status 400: invalid push payload: sessions[N].directory is required
write chunk: cloud: push chunk ...: status 400: invalid push payload: observations[N].title is required
It means a historical session mutation in sync_mutations is missing directory, a local sessions row included in the project export still has an empty/null directory, a local observations row included in the project export is missing a cloud-push required field, or a historical observation mutation is missing one of the required upsert fields: sync_id, session_id, type, title, content, or scope. Newer Engram versions write these fields correctly, but old journal rows, local session rows, or exported local observation rows may still need repair before first cloud upload.
Engram includes a temporary rescue helper. Despite the historical file name, it repairs missing session directories in both sync_mutations payloads and local sessions.directory rows, missing observation payload fields, and exported local observations rows whose required cloud-push fields are empty:
tools/repair-missing-session-directory.sh <project>Run it from inside the real project directory for session directory repairs. Observation repairs do not require a directory argument. Dry-run is the default.
cd /absolute/path/to/project
tools/repair-missing-session-directory.sh <project>Review the preview. For session repairs, confirm the detected Directory: is correct. For observation repairs, confirm the Local observation row is the authoritative local data. If an observation field such as title is still missing and the local row cannot fully infer it, preview an interactive repair first:
tools/repair-missing-session-directory.sh --interactive --seq 1677 sias-appThe interactive mode shows the mutation payload, matching local observation row when available, and a short content excerpt so you can provide only the missing required observation fields. Dry-run is still the default, so review the patched payload first. When it looks correct, rerun with --apply --interactive:
tools/repair-missing-session-directory.sh --apply --interactive --seq 1677 sias-appFor non-interactive repairs that can be inferred completely from local state, apply directly:
tools/repair-missing-session-directory.sh --apply <project>If doctor reveals another legacy blocker after each repair, use loop mode after you have reviewed a one-shot dry-run:
tools/repair-missing-session-directory.sh --apply --interactive --all <project>Loop mode repairs exactly one supported blocker (entity=session|observation op=upsert), reruns engram cloud upgrade doctor --project <project>, then repeats until doctor no longer reports a supported blocker. If doctor reports ready but local sessions rows included in the project export still have empty/null directory, loop mode applies that fallback repair and reruns doctor once more. If exported local observations rows are missing required fields such as title, loop mode applies that fallback repair next and reruns doctor again. It still stops on unsupported blockers, project mismatches, or observation data that cannot be fully inferred. In non-interactive loop mode, rerun with --interactive when the script asks for human-provided observation fields.
If one-shot mode finds no doctor blocker but reports local sessions with empty/null directory, preview and apply the fallback explicitly:
tools/repair-missing-session-directory.sh --fix-empty-sessions <project>
tools/repair-missing-session-directory.sh --apply --fix-empty-sessions <project>If final push fails with observations[N].title is required even though doctor is ready, preview and apply the exported observation fallback explicitly:
tools/repair-missing-session-directory.sh --fix-empty-observations <project>
tools/repair-missing-session-directory.sh --apply --fix-empty-observations <project>--fix-exported runs both exported-row fallbacks in one shot:
tools/repair-missing-session-directory.sh --fix-exported <project>
tools/repair-missing-session-directory.sh --apply --fix-exported <project>Use --max to cap the number of repairs and avoid accidental infinite loops. The default is 20:
tools/repair-missing-session-directory.sh --apply --interactive --all --max 5 <project>Then rerun the normal flow:
engram cloud upgrade doctor --project <project>
engram cloud upgrade repair --project <project> --dry-run
engram cloud upgrade repair --project <project> --apply
engram sync --cloud --project <project>For session repairs, the script patches one legacy row in sync_mutations by adding a JSON field:
"directory": "/absolute/path/to/project"It also updates sessions.directory only when the matching session row exists and its directory is empty. In the fallback path for sessions[N].directory is required, it updates only local sessions rows included in the requested project export scope where directory IS NULL OR directory = ''; it does not modify sync_mutations.
For observation repairs, the script reads the authoritative local row from observations using payload.sync_id or entity_key, then fills only missing or empty fields in the mutation payload:
sync_id, session_id, type, title, content, scope
It does not invent values in non-interactive mode. If the local observation row is missing, or the required payload fields would still be empty after patching, the script exits non-zero without modifying the database. Use --interactive when you need to provide missing observation values manually after reviewing the payload and content excerpt.
For exported local observation repairs, the script changes only observations rows that belong to the requested project export scope:
- rows where
ifnull(project, '') = '<project>' - rows with empty
projectwhosesession_idbelongs to asessions.project = '<project>'session
It does not touch sync_mutations. Safe inferred fields are filled as follows: empty sync_id becomes the row id, empty type becomes manual, empty title is inferred from non-empty content, and empty scope becomes project. Empty session_id is always blocked. Empty content is blocked unless you run with --interactive and provide it yourself.
It never changes last_acked_seq, never deletes mutations, and creates a timestamped database backup before each applied blocker. Loop mode can be noisy because it intentionally keeps one backup per repair.
If you do not pass --seq, the script runs:
engram cloud upgrade doctor --project <project>and extracts the first matching blocker:
seq=N entity=session op=upsert
seq=N entity=observation op=upsert
If you already know the sequence:
tools/repair-missing-session-directory.sh --seq 873 <project>
tools/repair-missing-session-directory.sh --apply --seq 873 <project>Loop mode does not accept --seq; it always asks doctor for the next supported blocker each iteration.
Precedence:
- Explicit directory argument.
git rev-parse --show-toplevelfrom the current directory.pwd.
The directory must be absolute. Good examples:
/home/user/work/sias-app
/Users/user/work/sias-app
C:/Users/user/work/sias-app
Bad example:
sias-app
On Windows/Git Bash, prefer forward slashes (C:/Users/user/work/sias-app) to avoid JSON and SQL escaping problems.
Use this when you are not currently inside the project directory:
tools/repair-missing-session-directory.sh --apply --seq 873 sias-app C:/Users/user/work/sias-appIf you want to inspect before using the helper:
sqlite3 ~/.engram/engram.db "select seq, entity, op, entity_key, payload from sync_mutations where seq = 873;"
sqlite3 ~/.engram/engram.db "select id, project, directory from sessions where id = 'manual-save-current';"
sqlite3 ~/.engram/engram.db "select id, project, started_at, directory from sessions where project = '<project>' and (directory is null or directory = '');"
sqlite3 ~/.engram/engram.db "select s.id, s.project, s.started_at, s.directory from sessions s where (s.directory is null or s.directory = '') and (s.project = '<project>' or s.id in (select session_id from observations where ifnull(project, '') = '<project>' union select session_id from user_prompts where ifnull(project, '') = '<project>'));"
sqlite3 ~/.engram/engram.db "select o.id, ifnull(o.sync_id, '') as sync_id, ifnull(o.session_id, '') as session_id, ifnull(o.project, '') as project, ifnull(o.type, '') as type, ifnull(o.title, '') as title, ifnull(o.scope, '') as scope from observations o where (ifnull(o.project, '') = '<project>' or (ifnull(o.project, '') = '' and o.session_id in (select id from sessions where ifnull(project, '') = '<project>'))) and (ifnull(o.sync_id, '') = '' or ifnull(o.session_id, '') = '' or ifnull(o.type, '') = '' or ifnull(o.title, '') = '' or ifnull(o.content, '') = '' or ifnull(o.scope, '') = '');"Do not manually edit SQLite without a backup.
Full symptom during engram sync --cloud:
write chunk: cloud: push chunk ...: status 400: invalid push payload: invalid character '\x1f' looking for beginning of value
Cause: the client sends the push body as a gzip-compressed envelope declared with Content-Type: application/vnd.engram.sync+gzip; version=1. \x1f is the first byte of the gzip magic number (0x1f 0x8b), so this error means the server tried to decode raw gzip bytes as plain JSON. That happens when a proxy in front of the cloud server dropped or rewrote the request Content-Type header, or when the running server binary predates the compressed envelope support.
Current server behavior: up-to-date engram cloud serve binaries sniff the gzip magic bytes and decode the compressed envelope when the declared Content-Type is missing or rewritten to a non-envelope value such as application/json, so a rewritten header no longer causes this error server-side. One shape still fails before sniffing: a request that still declares application/vnd.engram.sync+gzip with an unsupported version parameter is rejected with HTTP 415 (unsupported push payload), because the declared envelope version is validated before the request body is read. If you still see this error, the running server binary is older than this fix — upgrade the server.
Proxy advice: if you run a proxy (nginx, Traefik, an API gateway, a WAF) in front of the cloud server, configure it to preserve the client request Content-Type header — in particular application/vnd.engram.sync+gzip; version=1 — instead of replacing it with application/json or stripping it.
transport_failed is a wrapper around network, auth, server, or payload errors. Look for the concrete error message below it.
| Concrete error | Next step |
|---|---|
chunk_id does not match payload content hash |
Upgrade client and server to v1.14.8 or newer |
session payload directory is required |
Run the missing session directory helper |
sessions[N].directory is required |
Run the missing session directory helper with --fix-empty-sessions or --all |
observation payload title is required for upsert |
Run the missing session directory helper; it also repairs missing observation payload fields from local observations |
observations[N].title is required |
Run the missing session directory helper with --fix-empty-observations or --all |
401 or auth_required |
Check ENGRAM_CLOUD_TOKEN on the client and server |
403 or policy_forbidden |
Check the server-side ENGRAM_CLOUD_ALLOWED_PROJECTS policy for the denied project; a managed principal's project grant may also need checking. The client does not expose allowlist contents. |
server_unsupported |
Redeploy a cloud server with mutation endpoints |
| Message | Cause | Next step |
|---|---|---|
a managed admin already exists; refusing to create a duplicate first admin via CLI bootstrap |
A managed admin was already bootstrapped (via CLI or dashboard). | This is expected safety behavior, not a bug. Use the existing managed admin, or a documented recovery path, instead of re-running first-admin bootstrap. |
--issue-token requires ENGRAM_CLOUD_TOKEN_PEPPER to be configured |
--issue-token was passed without ENGRAM_CLOUD_TOKEN_PEPPER set. |
Set ENGRAM_CLOUD_TOKEN_PEPPER to a dedicated secret (distinct from ENGRAM_JWT_SECRET) and re-run. No admin/user was created by the failed attempt. |
stranded admin token recovery is not eligible |
recover-token found an unsafe admin or token state. |
Use engram cloud bootstrap recover-token only for the documented partial-bootstrap state. To replace one unused active bootstrap token whose output was lost, explicitly add --revoke-existing; it may be retried after another lost output only while the current token remains unused, and refuses used, revoked, ambiguous, or multiple-active-token states. |
recover-token requires ENGRAM_CLOUD_TOKEN_PEPPER to be configured |
The recovery command cannot hash a managed token safely. | Set the dedicated pepper, then retry. No token was created. |
connect cloud store |
ENGRAM_DATABASE_URL is missing/unreachable, same as any other engram cloud database command. |
Verify ENGRAM_DATABASE_URL and that Postgres is reachable, same as engram cloud serve. |
Rollback / legacy migration path: engram cloud bootstrap admin only adds new cloud_principals/cloud_human_users/cloud_principal_tokens/cloud_project_grants rows — it never disables ENGRAM_CLOUD_TOKEN or ENGRAM_CLOUD_ADMIN. If you want to stop using managed admins/tokens, simply keep using the legacy env credentials (or unset ENGRAM_CLOUD_TOKEN_PEPPER to disable managed-token authentication entirely); no migration or rollback command is required to fall back. See DOCS.md — Managed users, tokens, and CLI bootstrap for how engram cloud serve resolves managed tokens first, then legacy env-token credentials.
| Symptom | Cause | Next step |
|---|---|---|
A managed token returns 401 against /sync/* or /admin/* |
ENGRAM_CLOUD_TOKEN_PEPPER is not set on the server, the token was revoked, or the owning managed user is disabled. |
Confirm ENGRAM_CLOUD_TOKEN_PEPPER is set on the running engram cloud serve process (must match at issuance and at runtime) and that the token/user has not been revoked/disabled. |
A managed token returns 403 against /sync/* |
The managed principal has no project grant for the requested project (deny-by-default). | Grant the project via engram cloud bootstrap admin --grant-project <project> or the dashboard//admin/* grant routes. |
After any repair, verify in this order:
engram cloud status
engram cloud upgrade doctor --project <project>
engram cloud upgrade repair --project <project> --dry-run
engram cloud upgrade repair --project <project> --apply
engram sync --cloud --project <project>
engram sync --cloud --status --project <project>Expected result:
doctorno longer reports the same blocker.sync --cloudcompletes without canonicalization errors.last_acked_seqadvances.- Dashboard stats stop showing
0once data has been accepted by cloud.
- Do not delete
sync_mutationsrows to make the error disappear. - Do not edit
last_acked_seqmanually. - Do not invent a relative
directorylikesias-app. - Do not assume dashboard
0means local data is gone. - Do not run repair without a database backup.
- Cloud setup: Quickstart
- Full command reference: DOCS.md — Cloud CLI
- Autosync details: DOCS.md — Cloud Autosync