Use -Zpackage-workspace for packing

[hallipr]

• Add target to the workspace exclusions
• Add debug text for package version update scripts
• azure_template and azure_template_core should not match any released version

Pack-Crates.ps1

• Use the staged .crate file instead of the unpacked package folder in /target after a cargo publish --dry-run
  • https://rust-lang.zulipchat.com/#narrow/channel/246057-t-cargo/topic/Publishing.20crates.20from.20a.20monorepo.20at.20different.20periods/near/502267339
• Use +nightly -Zpackage-workspace instead of a vendor folder and powershell to order and stage unpublished dependencies
• Use --no-verify unless enabled with the -Verify switch
• Be more resilient to powershell Array / Single Item return values
• Create a publish API Json + Crate bundle for direct to api publishing

[Copilot]

Copilot reviewed 4 out of 7 changed files in this pull request and generated no comments.

Files not reviewed (3)

• eng/scripts/Pack-Crates.ps1: Language not supported
• eng/scripts/Update-PackageVersion.ps1: Language not supported
• eng/scripts/Yank-Crates.ps1: Language not supported

Comments suppressed due to low confidence (3)

eng/scripts/update-pathversions.rs:27

• The revised relative path for repo_root now excludes one level compared to the previous version; please verify that this change aligns with the intended repository structure.

let repo_root = script_root.join("../..").canonicalize()?;

eng/scripts/update-pathversions.rs:41

• [nitpick] Consider using a formal logging framework instead of println for production-level logging to better manage log verbosity and output.

println!("Processing {}", toml_file.path.display());

eng/pipelines/templates/jobs/pack.yml:35

• [nitpick] The removal of the dedicated Rust toolchain template could affect environment setup; please ensure that the correct Rust version is still being applied in the build pipeline.

  - template: /eng/pipelines/templates/steps/use-rust.yml@self

[hallipr]

/azp run

[azure-pipelines]

You have several pipelines (over 10) configured to build pull requests in this repository. Specify which pipelines you would like to run by using /azp run [pipelines] command. You can specify multiple pipelines using a comma separated list.

[heaths]

/azp run rust - template

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[heaths]

What's not clear from the PR description is why. Why are we doing all this work - often redoing what cargo pack and/or cargo publish would I'm really not crazy about the cost of owning all this and keeping it in sync with any changes that the cargo team might make. They don't really document what's in a .crate file which means it could change and all our stuff breaks.

I'd really like to understand why we're doing all these changes.

[heaths]

This is always ignored. Why did you add it? This is going to raise a lot of eyebrows.

[hallipr]

it's not ignored 😬. The targets folder will get package folder in it. If you run any cargo commands on the packages in /targets, it complains about the package thinking its running in a workspace but not being a member or excluded

[heaths]

Why are you running commands against anything targets? That's not our place space. For package analysis - though I maintain there's nothing to analyze (it's zipped source) - that should probably be done outside the source repo in the temp working dir AzDO, GHA, etc. give us. Let's not unnecessarily complicate things.

[heaths]

Why not just use Get-FileHash? It's been around forever. Best not to reinvent the wheel and own its maintenance.

[hallipr]

this isn't computing a hash. Its constructing a binary file for posting to the crates.io api

[heaths]

...which comes back to my original question: why are we doing that? I get why we crack open .nupkg files: there's binaries to check for signing, viruses, etc. .crate files are zipped source. We spending so much time on redoing what cargo publish already does which not only prevents some other work from getting done, but I worry creates a dependency on undocumented implementation details we'll forever have to maintain ourselves.

[hallipr]

I'd really like to understand why we're doing all these changes.

@epage said that we can only rely on the crate file in target and not the package folder

https://rust-lang.zulipchat.com/#narrow/channel/246057-t-cargo/topic/Publishing.20crates.20from.20a.20monorepo.20at.20different.20periods/near/502267339

Note that any files in target/ made by cargo publrsh are most likely considered incidental and not subject to compatibility guarentees. cargo package would only guarentee .crate. We are planning to move the uncompressed folder.

So our release process needs to be .crate centric. Also, by running cargo publish on top of the pre-normalized cargo.toml inside the crate file, we end up with a normalized cargo.toml.orig file instead of a normalized cargo.toml and normal looking cargo.toml.orig

[hallipr]

Also, it feels like maintaining the API call will be less maintenance than all of the powershell we have around orchestrating the dependencies during build.

[heaths]

All the powershell around dependencies was necessary, though, because monorepos aren't well-supported by cargo. I still don't understand the point of us pulling apart a .crate file to "analyze" it, reconstruct it, and call crates.io APIs directly ourselves when cargo publish is right there.

What are we even analyzing for a zipped source archive? Anything we could scan for we should against the source itself during PRs, CIs, etc. All that can already be done and not at the last minute when we're trying to publish.

/cc @RickWinter

I think we should leave this PR as-is for now while we discuss. I know there are some other high-pri EngSys work items we need to address yet e.g., caching, whether Azure Pipelines' caching is good enough or if we should use sccache.

[hallipr]

when cargo publish is right there.

cargo publish is only right there if we publish an uncompressed folder. We only get an uncompressed folder by checking out the repo or by decompressing the .tgz file. We can't checkout the repo in the deployment job.

https://eng.ms/docs/cloud-ai-platform/devdiv/one-engineering-system-1es/1es-docs/1es-pipeline-templates/migration-release/value-proposition

https://eng.ms/docs/cloud-ai-platform/devdiv/one-engineering-system-1es/1es-docs/1es-pipeline-templates/changemanagement/checkout-in-custom-release

Hence, 1ES PT does not support checkout in release jobs.

[hallipr]

Because we use workspace references, our package folders can't be published in isolation. The contents of the tarball are publishable in isolation and the only effect of "double publishing" the tarball is the mangling of the Cargo.toml.orig file.

If we publish the original .tgz with a direct api call, we publish with the original Cargo.toml in Cargo.toml.orig.

[heaths]

Until we can get toolset changes, I think we need to seek an exemption. That's not how cargo works and we're relying solely on observed behavior to make critical publishing decisions.

[hallipr]

I'm documenting all of the current dependency version and path behaviors in issue #2475 "Document dependency requirements".