Skip to content

fix: migrate env-var substitution to serde-saphyr properties - #2868

Closed
CommanderStorm wants to merge 3 commits into
maplibre:mainfrom
CommanderStorm:saphyr-properties-migration
Closed

fix: migrate env-var substitution to serde-saphyr properties#2868
CommanderStorm wants to merge 3 commits into
maplibre:mainfrom
CommanderStorm:saphyr-properties-migration

Conversation

@CommanderStorm

Copy link
Copy Markdown
Member

Fixes #2851.

Summary

  • Replaces subst::substitute's text-level preprocessing — which scanned comments and quoted strings — with serde-saphyr's properties feature, which only interpolates inside plain YAML scalar values. Comments containing ${VAR} no longer abort startup.
  • Adopts PropertySyntax::BracedOrBare so existing configs using ${VAR}, ${VAR:-default}, and bare $VAR keep working unchanged.
  • Drops subst from the workspace. Pins serde-saphyr to the master rev 261ccbe6… containing the four PRs we contributed upstream (chore: update dependencies #122 defaults, chore(deps): bump serde_json from 1.0.57 to 1.0.58 #125 modifiers, chore(deps): bump actix-cors from 0.2.0 to 0.4.1 #128 error-on-missing, chore(deps): bump serde_json from 1.0.58 to 1.0.59 #129 bare syntax). Allowed via deny.toml; both have TODOs to drop once 0.0.28 is published.
  • Substitution failures now flow through ConfigFileError::YamlParseError with a wrapper that emits martin::config::substitution as the diagnostic code (preserved from the old dedicated variant) when the underlying saphyr error is property-related, otherwise martin::config::yaml. Inline insta snapshots across the failing-deserializer tests were updated to match the consistent code/help/url footer.

Behavior change to flag in release notes: substitution no longer happens inside quoted or block scalars. A YAML like connection_string: \"\${DATABASE_URL}\" now keeps the literal \${DATABASE_URL} instead of expanding. Drop the quotes (or use connection_string: \${DATABASE_URL}) to get the previous behavior.

Test plan

  • cargo test -p martin --lib --no-default-features --features postgres,pmtiles — 212/212 pass
  • Added regression test substitution_ignores_dollar_tokens_in_comments using the exact YAML from Martin tries to substitute environment variables in config comments #2851
  • cargo fmt --all --check clean
  • CI: full feature matrix via cargo-hack
  • CI: cargo-deny (the new allow-git entry is the load-bearing change)
  • CI: clippy -D warnings workspace-wide

…e#2851)

Replaces `subst::substitute`'s text-level preprocessing — which scanned
comments and quoted strings — with `serde-saphyr`'s `properties` feature
that only interpolates inside plain YAML scalar values. Adopts
`PropertySyntax::BracedOrBare` so existing configs using `${VAR}`,
`${VAR:-default}`, and bare `$VAR` keep working unchanged.

Drops `subst` from the workspace. Pins `serde-saphyr` to the master rev
containing PRs maplibre#122/maplibre#125/maplibre#128/maplibre#129 (defaults, modifiers, error-on-missing
forms, bare syntax) until 0.0.28 is published; tracked in deny.toml.

Substitution failures now flow through `ConfigFileError::YamlParseError`
with a wrapper that emits `martin::config::substitution` as the diagnostic
code (preserved from the old dedicated variant) when the underlying
saphyr error is an `UnresolvedProperty` / `InvalidPropertyName` /
`PropertyRequiredButUnset` / `PropertyRequiredButEmpty`. Updated
inline insta snapshots across the failing-deserializer tests so all YAML
errors render consistent code/help/url footers.
Copilot AI review requested due to automatic review settings June 10, 2026 19:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

pre-commit-ci Bot and others added 2 commits June 10, 2026 19:59
Clippy's `doc_markdown` lint (enabled via workspace pedantic) requires
URLs in doc comments to be enclosed in angle brackets or rendered as
Markdown links.
@CommanderStorm

Copy link
Copy Markdown
Member Author

Update — integration tests surfaced a subst-specific syntax gap

Three CI groups are red. Two have the same root cause; one is the cargo-publish dry-run which is expected.

Test on ubuntu-latest, Test postgis:18-3.6 sslmode=require, Build and test docker images

All three fail in the same place:

ERROR Failed to create postgres pool: Unable to parse connection string
${DATABASE_URL:postgres://postgres@localhost:5432/db}: invalid connection string

The fixtures (tests/config.yaml:48, tests/config-process.yaml:12) use subst's single-colon default syntax inside single quotes:

connection_string: '${DATABASE_URL:postgres://postgres@localhost:5432/db}'

subst accepts this — its docs say:

Variables have the form $NAME, ${NAME} or ${NAME:default}.
(subst-0.3.8 lib.rs:103)

POSIX shell, docker-compose, and saphyr's properties all use ${VAR:-default} (with the -). Saphyr would normally raise InvalidPropertyName for the single-colon form, but in these fixtures the string is single-quoted, and saphyr only interpolates plain scalars — so the placeholder reaches tokio-postgres verbatim and fails as an invalid connection string. Two behaviour deltas from subst, stacked.

Three options to consider:

  1. Upstream saphyr — add ${NAME:default} as an alias for ${NAME:-default} (the docker-compose form). Easiest on downstreams migrating off subst, but unblocking this PR depends on an external review/release.
  2. Rewrite the fixtures + a release note — change every '${VAR:default}' to unquoted ${VAR:-default}. Breaking for users with the old syntax in their configs.
  3. Tiny pre-processor in Martin — before handing the YAML to saphyr, rewrite ${VAR:default}${VAR:-default} in plain and quoted scalars. Keeps user configs compatible, no upstream dependency.

I lean (1) → (3) → (2). Want me to open the saphyr issue/PR? I already drafted the body, but filing an issue under your identity on an external repo needs your sign-off.

Test publishing via cargo-publish

Expected to be red until 0.0.28 is on crates.io — cargo publish --dry-run can't verify a git dep. Same TODO already tracked in Cargo.toml and deny.toml.

Pushed: clippy fix

fc766f37 wraps the bare issue URL in the new regression test in <…> so clippy::doc_markdown passes.

@CommanderStorm

Copy link
Copy Markdown
Member Author

ups, left claude unatented for a bit 👀

@github-actions

github-actions Bot commented Jun 10, 2026

Copy link
Copy Markdown

Performance Comparison mainsaphyr-properties-migration

Total Elapsed Time: 37.19s → 38.68s (+4.0%)
CPU Baseline: 61.95µs → 60.79µs (-1.9%)
Benchmark ID: timing

timing - Function execution time metrics.

+----------------------------+------------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| Function                   | Calls                        | Avg                              | P95                              | Total                            | % Total                      |
+----------------------------+------------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| martin::main               | 1 → 1 (+0.0%)                | 37.20s → 38.67s (+4.0%)          | 37.21s → 38.69s (+4.0%)          | 37.19s → 38.68s (+4.0%)          | 100.00% → 100.00% (+0.0%)    |
+----------------------------+------------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| martin::start              | 1 → 1 (+0.0%)                | 37.20s → 38.67s (+4.0%)          | 37.21s → 38.69s (+4.0%)          | 37.19s → 38.68s (+4.0%)          | 99.99% → 99.99% (+0.0%)      |
+----------------------------+------------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| content::get_tile          | 2200600 → 2200600 (+0.0%)    | 13.02µs → 12.39µs (-4.8%)        | 15.41µs → 15.17µs (-1.6%)        | 28.65s → 27.27s (-4.8%)          | 77.05% → 70.50% (-8.5%)      |
+----------------------------+------------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| content::get_http_response | 2200600 → 2200600 (+0.0%)    | 9.21µs → 8.68µs (-5.8%)          | 12.34µs → 12.29µs (-0.4%)        | 20.27s → 19.09s (-5.8%)          | 54.50% → 49.35% (-9.4%)      |
+----------------------------+------------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| content::get_tile_content  | 2200600 → 2200600 (+0.0%)    | 8.20µs → 7.67µs (-6.5%)          | 11.18µs → 11.09µs (-0.8%)        | 18.04s → 16.87s (-6.5%)          | 48.50% → 43.61% (-10.1%)     |
+----------------------------+------------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| content::new               | 2200600 → 2200600 (+0.0%)    | 1.47µs → 1.41µs (-4.1%)          | 1.15µs → 1.00µs (-13.0%)         | 3.22s → 3.10s (-3.7%)            | 8.67% → 8.00% (-7.7%)        |
+----------------------------+------------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| content::recompress        | 2200600 → 2200600 (+0.0%)    | 811.00ns → 602.00ns (-25.8%) 🚀  | 80.00ns → 150.00ns (+87.5%) ⚠️   | 1.76s → 1.29s (-26.7%) 🚀        | 4.72% → 3.34% (-29.2%) 🚀    |
+----------------------------+------------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| source::get_sources        | 2200600 → 2200600 (+0.0%)    | 654.00ns → 548.00ns (-16.2%)     | 922.00ns → 821.00ns (-11.0%)     | 1.44s → 1.21s (-16.0%)           | 3.87% → 3.12% (-19.4%)       |
+----------------------------+------------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| content::encode            | 100100 → 100100 (+0.0%)      | 16.57µs → 12.01µs (-27.5%) 🚀    | 28.48µs → 11.43µs (-59.9%) 🚀    | 1.66s → 1.20s (-27.7%) 🚀        | 4.46% → 3.11% (-30.3%) 🚀    |
+----------------------------+------------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| server::new_server         | 1 → 1 (+0.0%)                | 291.71µs → 216.51µs (-25.8%) 🚀  | 291.84µs → 216.57µs (-25.8%) 🚀  | 291.82µs → 216.57µs (-25.8%) 🚀  | 0.00% → 0.00% (+0.0%)        |
+----------------------------+------------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| optimizer::encode          | 1 → 1 (+0.0%)                | 63.12µs → 52.37µs (-17.0%)       | 63.13µs → 52.38µs (-17.0%)       | 63.13µs → 52.37µs (-17.0%)       | 0.00% → 0.00% (+0.0%)        |
+----------------------------+------------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| optimizer::encode_into     | 4 → 4 (+0.0%)                | 7.85µs → 7.38µs (-6.0%)          | 16.66µs → 20.27µs (+21.7%) ⚠️    | 31.41µs → 29.51µs (-6.0%)        | 0.00% → 0.00% (+0.0%)        |
+----------------------------+------------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| encode::write_to           | 4 → 4 (+0.0%)                | 6.14µs → 6.74µs (+9.8%)          | 15.12µs → 18.78µs (+24.2%) ⚠️    | 24.58µs → 26.96µs (+9.7%)        | 0.00% → 0.00% (+0.0%)        |
+----------------------------+------------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| writer::with               | 72 → 72 (+0.0%)              | 123.00ns → 110.00ns (-10.6%)     | 251.00ns → 370.00ns (+47.4%) ⚠️  | 8.92µs → 7.94µs (-11.0%)         | 0.00% → 0.00% (+0.0%)        |
+----------------------------+------------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+
| optimizer::analyze         | 1 → 1 (+0.0%)                | 12.22µs → 7.48µs (-38.8%) 🚀     | 12.22µs → 7.48µs (-38.8%) 🚀     | 12.22µs → 7.48µs (-38.8%) 🚀     | 0.00% → 0.00% (+0.0%)        |
+----------------------------+------------------------------+----------------------------------+----------------------------------+----------------------------------+------------------------------+

alloc-bytes - Exclusive allocation bytes by each function (excluding nested calls).

+--------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+----------------------------+
| Function                       | Calls                        | Avg                            | P95                            | Total                          | % Total                    |
+--------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+----------------------------+
| content::get_tile_content      | 2200600 → 2200600 (+0.0%)    | 93.0 KB → 93.0 KB (+0.0%)      | 181.9 KB → 181.9 KB (+0.0%)    | 195.2 GB → 195.2 GB (+0.0%)    | 84.06% → 84.06% (+0.0%)    |
+--------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+----------------------------+
| content::encode                | 100100 → 100100 (+0.0%)      | 344.5 KB → 344.5 KB (+0.0%)    | 347.2 KB → 347.2 KB (+0.0%)    | 32.9 GB → 32.9 GB (+0.0%)      | 14.16% → 14.16% (+0.0%)    |
+--------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+----------------------------+
| source::get_sources            | 2200600 → 2200600 (+0.0%)    | 1.5 KB → 1.5 KB (+0.0%)        | 2.4 KB → 2.4 KB (+0.0%)        | 3.1 GB → 3.1 GB (+0.0%)        | 1.35% → 1.35% (+0.0%)      |
+--------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+----------------------------+
| content::get_tile              | 2200600 → 2200600 (+0.0%)    | 297 B → 297 B (+0.0%)          | 296 B → 296 B (+0.0%)          | 624.7 MB → 624.7 MB (+0.0%)    | 0.26% → 0.26% (+0.0%)      |
+--------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+----------------------------+
| content::get_http_response     | 2200600 → 2200600 (+0.0%)    | 182 B → 182 B (+0.0%)          | 200 B → 200 B (+0.0%)          | 383.6 MB → 383.6 MB (+0.0%)    | 0.16% → 0.16% (+0.0%)      |
+--------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+----------------------------+
| martin::start                  | 1 → 1 (+0.0%)                | 3.0 MB → 3.0 MB (+0.0%)        | 3.0 MB → 3.0 MB (+0.0%)        | 3.0 MB → 3.0 MB (+0.0%)        | 0.00% → 0.00% (+0.0%)      |
+--------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+----------------------------+
| martin::main                   | 1 → 1 (+0.0%)                | 139.9 KB → 141.9 KB (+1.4%)    | 139.9 KB → 142.0 KB (+1.5%)    | 139.9 KB → 141.9 KB (+1.4%)    | 0.00% → 0.00% (+0.0%)      |
+--------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+----------------------------+
| optimizer::encode              | 1 → 1 (+0.0%)                | 67.7 KB → 67.7 KB (+0.0%)      | 67.7 KB → 67.7 KB (+0.0%)      | 67.7 KB → 67.7 KB (+0.0%)      | 0.00% → 0.00% (+0.0%)      |
+--------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+----------------------------+
| server::new_server             | 1 → 1 (+0.0%)                | 31.1 KB → 31.1 KB (+0.0%)      | 31.1 KB → 31.1 KB (+0.0%)      | 31.1 KB → 31.1 KB (+0.0%)      | 0.00% → 0.00% (+0.0%)      |
+--------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+----------------------------+
| tile::from_tile                | 4 → 4 (+0.0%)                | 376 B → 376 B (+0.0%)          | 376 B → 376 B (+0.0%)          | 1.5 KB → 1.5 KB (+0.0%)        | 0.00% → 0.00% (+0.0%)      |
+--------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+----------------------------+
| encode::write_to               | 4 → 4 (+0.0%)                | 370 B → 370 B (+0.0%)          | 624 B → 624 B (+0.0%)          | 1.4 KB → 1.4 KB (+0.0%)        | 0.00% → 0.00% (+0.0%)      |
+--------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+----------------------------+
| writer::with                   | 72 → 72 (+0.0%)              | 16 B → 16 B (+0.0%)            | 380 B → 380 B (+0.0%)          | 1.2 KB → 1.2 KB (+0.0%)        | 0.00% → 0.00% (+0.0%)      |
+--------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+----------------------------+
| encode::dict_may_be_beneficial | 4 → 4 (+0.0%)                | 256 B → 256 B (+0.0%)          | 256 B → 256 B (+0.0%)          | 1.0 KB → 1.0 KB (+0.0%)        | 0.00% → 0.00% (+0.0%)      |
+--------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+----------------------------+
| sort::sort                     | 4 → 4 (+0.0%)                | 120 B → 120 B (+0.0%)          | 160 B → 160 B (+0.0%)          | 480 B → 480 B (+0.0%)          | 0.00% → 0.00% (+0.0%)      |
+--------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+----------------------------+
| writer::write_header           | 4 → 4 (+0.0%)                | 69 B → 69 B (+0.0%)            | 104 B → 104 B (+0.0%)          | 276 B → 276 B (+0.0%)          | 0.00% → 0.00% (+0.0%)      |
+--------------------------------+------------------------------+--------------------------------+--------------------------------+--------------------------------+----------------------------+

Threads

Total Alloc: 3.0 MB → 3.0 MB (+0.0%)
Total Dealloc: 301.4 MB → 301.4 MB (+0.0%)
Mem Diff: -298.5 MB → -298.5 MB (+0.0%)

+-------------+------------------------------+------------------------------+------------------------------+------------------------------+---------------------------------+
| Thread      | CPU % Avg                    | CPU % Max                    | Alloc                        | Dealloc                      | Mem Diff                        |
+-------------+------------------------------+------------------------------+------------------------------+------------------------------+---------------------------------+
| martin      | 0.10% → 0.10% (+0.0%)        | 8.00% → 8.00% (+0.0%)        | 2.8 MB → 2.8 MB (+0.0%)      | 1.8 MB → 1.9 MB (+5.6%)      | 1019.1 KB → 923.9 KB (-9.3%)    |
+-------------+------------------------------+------------------------------+------------------------------+------------------------------+---------------------------------+
| hp-mcp      | 0.00% → 0.00% (+0.0%)        | 0.00% → 0.00% (+0.0%)        | 54.7 KB → 54.7 KB (+0.0%)    | 3.4 KB → 3.4 KB (+0.0%)      | 51.4 KB → 51.4 KB (+0.0%)       |
+-------------+------------------------------+------------------------------+------------------------------+------------------------------+---------------------------------+
| 🆕 hp-debug | 0.00% → 11.30% (+100.0%) ⚠️  | 0.00% → 31.90% (+100.0%) ⚠️  | 0 B → 10.3 KB (+100.0%) ⚠️   | 0 B → 299.5 MB (+100.0%) ⚠️  | 0 B → -299.5 MB (+100.0%) ⚠️    |
+-------------+------------------------------+------------------------------+------------------------------+------------------------------+---------------------------------+

Generated with hotpath-rs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Martin tries to substitute environment variables in config comments

2 participants