Skip to content

Add shopify validate command for offline validation#8142

Open
craigmichaelmartin wants to merge 12 commits into
mainfrom
feature/shopify-validate-command
Open

Add shopify validate command for offline validation#8142
craigmichaelmartin wants to merge 12 commits into
mainfrom
feature/shopify-validate-command

Conversation

@craigmichaelmartin

Copy link
Copy Markdown
Contributor

What

Adds a new top-level shopify validate command that ports the four validation scripts from ai-toolkit-source's shopify-dev-tools package into first-class CLI subcommands for offline, deterministic validation:

Subcommand Validates Engine
shopify validate theme Liquid/theme code (full-app + stateless codeblock modes) @shopify/theme-check-* (already shipped by the CLI)
shopify validate graphql GraphQL operations against bundled Admin/Storefront/Customer/Partner/Payments schemas; reports required offline access scopes graphql + bundled introspection JSON
shopify validate functions Shopify Functions input queries shared GraphQL engine
shopify validate components UI-extension component code (TSX/JSX/HTML) virtual TypeScript environment + bundled @shopify/ui-extensions type defs

How

All four subcommands sit on one shared engine under packages/cli/src/cli/services/validate/engine/:

  • A dependency-free result contract (ValidationResult const-union, ValidationResponse) and a barrel-decoupled formatter — so the non-component validators never pull in the TypeScript compiler.
  • Pure, injectable version resolution over a bundled version catalog.
  • One lazy gunzip walk-up data-loader that resolves bundled assets in dev (tsx/vitest), the split esbuild bundle, and the published /assets alike.
  • Pure-core + thin-orchestrator split per subcommand (command parses flags → service holds all logic → renders/records/exits).

Behavior preserved from the source tools (flags, --json shape {success, responses, resolvedVersion?}, exit codes, stdin), while adapting to CLI conventions: cli-kit rendering, outputResult + AbortSilentError for --json, telemetry via addPublicMetadata (new cmd_validate_* fields). Agent-only plumbing (random-UUID artifact IDs, shopify.dev network instrumentation) is dropped for deterministic output. Bad --file/empty stdin now return a structured FAILED payload instead of crashing.

Reference data (~13.7 MB compressed: GraphQL schemas + UI-extension .d.ts.gz) is bundled for fully offline validation and copied into dist/ via a recursive esbuild rule. typescript becomes a lazily-loaded, esbuild-external runtime dependency (used only by validate components).

Test plan

  • nx type-check cli ✅ · nx lint cli
  • nx vitest cli for the validate suites: 81/81 pass (real temp dirs, no fs mocking)
  • nx bundle cli ✅ — bundled binary smoke-tested for all four subcommands:
    • validate graphql --api admin --code 'query { shop { name } }'success, exit 0
    • validate graphql with an invalid field → failed, exit 1 (Cannot query field "nope" on type "Shop")
    • validate functions --api functions_discount (valid input query) → success
    • validate theme (codeblock) → success
    • validate components --api polaris-app-home --code '<s-text>hello</s-text>' --language htmlsuccess, validatedComponents: ["s-text"]
  • Regenerated oclif.manifest.json, README.md, and dev-docs; changeset added.

Notes / reviewer callouts

  • Package size: adds ~13.7 MB of compressed schema/type data (Admin schema alone ~4.8 MB). This is the faithful-offline tradeoff; the loader decompresses lazily per requested version, so it is on-disk weight only. Happy to trim historical Admin versions if preferred.
  • New runtime dep: typescript (lazy, esbuild-external) for validate components.
  • A scoped packages/cli/assets/validate/.gitignore re-includes the vendored type packages' dist/ folders (otherwise the repo-wide dist ignore would drop ~320 .d.ts.gz reference files).

🤖 Generated with Claude Code

craigmichaelmartin and others added 6 commits July 21, 2026 12:29
Consolidate the 20 Shopify-AI-Toolkit skills into a single
.agents/skills/shopify/SKILL.md. The description matches liberally
across every Shopify surface (Admin/Storefront/Customer/Partner/Payments
GraphQL, custom data, Functions, Hydrogen, Liquid, Polaris + UI
extensions, onboarding, CLI ops, UCP, App Store review, docs search)
within the 1024-char limit.

The body makes two steps mandatory and non-skippable: search dev docs
first via `shopify doc search`, then validate the final outcome via
`shopify validate [topic]`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address review feedback: be explicit that `shopify commands` lists the
available commands and `--help` at any level describes
commands/subcommands/flags, so the agent confirms the real surface
(including `shopify validate`) instead of guessing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Distinguish two outcomes in the validate step: a chat-reply example must
be a complete, fully valid artifact (no snippets or placeholder syntax)
and validated via a temp file; a filesystem change must validate every
touched file with the appropriate topic, not just the main one.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude Code 2.1.86 caps skill descriptions at 250 chars in the /skills
listing (to reduce context), so the previous 995-char description was
truncated mid-list — ~12 of 20 surfaces (Hydrogen, themes, extensions,
onboarding, CLI, UCP, App Store review, docs) fell past the cut and
never reached the matcher. Replace it with a 248-char gateway that names
every surface up front and survives the cap.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address review feedback: the helper-reuse note should point to
@shopify/cli, not @shopify/cli-kit.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Migrate the four validation scripts from ai-toolkit-source's
shopify-dev-tools package into first-class CLI subcommands:

- `shopify validate theme`      Liquid/theme code via @shopify/theme-check-*
- `shopify validate graphql`    GraphQL operations against bundled API schemas
                                (+ required offline access scopes)
- `shopify validate functions`  Shopify Functions input queries
- `shopify validate components` UI-extension component code (TSX/JSX/HTML)
                                type-checked in a virtual TypeScript environment

All four share one dependency-free engine (result contract, formatter,
pure version resolution, and a lazy gunzip walk-up data-loader) under
packages/cli/src/cli/services/validate/engine. The barrel-decoupled
contract keeps the TypeScript compiler out of the non-component
validators. Reference schemas and type definitions are bundled for fully
offline, deterministic validation. Telemetry rides cli-kit's
addPublicMetadata via new cmd_validate_* fields; the source scripts'
agent-only plumbing (artifact IDs, network instrumentation) is dropped.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@craigmichaelmartin
craigmichaelmartin requested review from a team as code owners July 21, 2026 18:46
@binks-code-reviewer

Copy link
Copy Markdown

🤖 Code Review · Skipped — PR has too many files for review (limit: 1000)

@github-actions github-actions Bot added the Area: @shopify/cli @shopify/cli package issues label Jul 21, 2026
craigmichaelmartin and others added 3 commits July 21, 2026 14:49
The doc-search --api-name flag is free-form: valid values are defined by
shopify.dev (not the CLI), documented examples are admin/storefront/
hydrogen/functions, and unrecognized values are silently ignored (no
error, unlike --api-version). Stop inventing values (customer/partner/
payments/liquid/polaris), tell the agent to omit it when unsure, and fix
the illustrative polaris -> app-home.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Model omitting --api-name when the value isn't a documented one,
consistent with the clarified guidance.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add unified Shopify umbrella skill
@craigmichaelmartin

Copy link
Copy Markdown
Contributor Author

/snapit

@github-actions

Copy link
Copy Markdown
Contributor

🫰✨ Thanks @craigmichaelmartin! Your snapshot has been published to npm.

Test the snapshot by installing your package globally:

pnpm i -g --@shopify:registry=https://registry.npmjs.org @shopify/cli@0.0.0-snapshot-20260721185120

Caution

After installing, validate the version by running shopify version in your terminal.
If the versions don't match, you might have multiple global instances installed.
Use which shopify to find out which one you are running and uninstall it.

craigmichaelmartin and others added 3 commits July 21, 2026 15:05
List the full set of shopify.dev doc-search API names in the "search the
docs first" step so agents pass only valid --api-name values instead of
inventing them: admin, admin-extensions, checkout-ui-extensions,
customer-account-ui-extensions, pos-ui-extensions, app-home, storefront,
partner, customer, payments-apps, hydrogen, liquid,
storefront-web-components, functions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Undo the changes from 80bb226 ("Clarify --api-name is an optional,
unvalidated hint") and the enumeration layered on top of it, restoring
the prior wording: the inline example list in the doc-search snippet, the
simple "scope to a surface / omit to search all" bullet, the Themes
--api-name liquid example, the polaris label, and the original gotcha.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
List the full set of shopify.dev doc-search API names in the "search the
docs first" step, phrased as a plain enumeration without dismissive
framing: admin, admin-extensions, checkout-ui-extensions,
customer-account-ui-extensions, pos-ui-extensions, app-home, storefront,
partner, customer, payments-apps, hydrogen, liquid,
storefront-web-components, functions. Also correct the illustrative
snippet's `payments` to `payments-apps`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Differences in type declarations

We detected differences in the type declarations generated by Typescript for this branch compared to the baseline ('main' branch). Please, review them to ensure they are backward-compatible. Here are some important things to keep in mind:

  • Some seemingly private modules might be re-exported through public modules.
  • If the branch is behind main you might see odd diffs, rebase main into this branch.

New type declarations

We found no new type declarations in this PR

Existing type declarations

packages/cli-kit/dist/public/node/metadata.d.ts
@@ -34,7 +34,7 @@ export type SensitiveSchema<T> = T extends RuntimeMetadataManager<infer _TPublic
  * @returns A container for the metadata.
  */
 export declare function createRuntimeMetadataContainer<TPublic extends AnyJson, TSensitive extends AnyJson = Record<string, never>>(defaultPublicMetadata?: Partial<TPublic>): RuntimeMetadataManager<TPublic, TSensitive>;
-type CmdFieldsFromMonorail = PickByPrefix<MonorailEventPublic, 'cmd_all_'> & PickByPrefix<MonorailEventPublic, 'cmd_app_'> & PickByPrefix<MonorailEventPublic, 'cmd_create_app_'> & PickByPrefix<MonorailEventPublic, 'cmd_theme_'> & PickByPrefix<MonorailEventPublic, 'store_'> & PickByPrefix<MonorailEventPublic, 'env_auto_upgrade_'>;
+type CmdFieldsFromMonorail = PickByPrefix<MonorailEventPublic, 'cmd_all_'> & PickByPrefix<MonorailEventPublic, 'cmd_app_'> & PickByPrefix<MonorailEventPublic, 'cmd_create_app_'> & PickByPrefix<MonorailEventPublic, 'cmd_theme_'> & PickByPrefix<MonorailEventPublic, 'cmd_validate_'> & PickByPrefix<MonorailEventPublic, 'store_'> & PickByPrefix<MonorailEventPublic, 'env_auto_upgrade_'>;
 declare const coreData: RuntimeMetadataManager<CmdFieldsFromMonorail, {
     commandStartOptions: {
         startTime: number;
packages/cli-kit/dist/public/node/monorail.d.ts
@@ -77,6 +77,11 @@ export interface Schemas {
             cmd_app_validate_valid?: Optional<boolean>;
             cmd_app_validate_issue_count?: Optional<number>;
             cmd_app_validate_file_count?: Optional<number>;
+            cmd_validate_subcommand?: Optional<string>;
+            cmd_validate_result?: Optional<string>;
+            cmd_validate_api?: Optional<string>;
+            cmd_validate_api_version?: Optional<string>;
+            cmd_validate_json?: Optional<boolean>;
             cmd_dev_tunnel_type?: Optional<string>;
             cmd_dev_tunnel_custom_hash?: Optional<string>;
             cmd_dev_urls_updated?: Optional<boolean>;

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area: @shopify/cli @shopify/cli package issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant