Add shopify validate command for offline validation#8142
Open
craigmichaelmartin wants to merge 12 commits into
Open
Add shopify validate command for offline validation#8142craigmichaelmartin wants to merge 12 commits into
shopify validate command for offline validation#8142craigmichaelmartin wants to merge 12 commits into
Conversation
Consolidate the 20 Shopify-AI-Toolkit skills into a single .agents/skills/shopify/SKILL.md. The description matches liberally across every Shopify surface (Admin/Storefront/Customer/Partner/Payments GraphQL, custom data, Functions, Hydrogen, Liquid, Polaris + UI extensions, onboarding, CLI ops, UCP, App Store review, docs search) within the 1024-char limit. The body makes two steps mandatory and non-skippable: search dev docs first via `shopify doc search`, then validate the final outcome via `shopify validate [topic]`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address review feedback: be explicit that `shopify commands` lists the available commands and `--help` at any level describes commands/subcommands/flags, so the agent confirms the real surface (including `shopify validate`) instead of guessing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Distinguish two outcomes in the validate step: a chat-reply example must be a complete, fully valid artifact (no snippets or placeholder syntax) and validated via a temp file; a filesystem change must validate every touched file with the appropriate topic, not just the main one. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude Code 2.1.86 caps skill descriptions at 250 chars in the /skills listing (to reduce context), so the previous 995-char description was truncated mid-list — ~12 of 20 surfaces (Hydrogen, themes, extensions, onboarding, CLI, UCP, App Store review, docs) fell past the cut and never reached the matcher. Replace it with a 248-char gateway that names every surface up front and survives the cap. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address review feedback: the helper-reuse note should point to @shopify/cli, not @shopify/cli-kit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Migrate the four validation scripts from ai-toolkit-source's
shopify-dev-tools package into first-class CLI subcommands:
- `shopify validate theme` Liquid/theme code via @shopify/theme-check-*
- `shopify validate graphql` GraphQL operations against bundled API schemas
(+ required offline access scopes)
- `shopify validate functions` Shopify Functions input queries
- `shopify validate components` UI-extension component code (TSX/JSX/HTML)
type-checked in a virtual TypeScript environment
All four share one dependency-free engine (result contract, formatter,
pure version resolution, and a lazy gunzip walk-up data-loader) under
packages/cli/src/cli/services/validate/engine. The barrel-decoupled
contract keeps the TypeScript compiler out of the non-component
validators. Reference schemas and type definitions are bundled for fully
offline, deterministic validation. Telemetry rides cli-kit's
addPublicMetadata via new cmd_validate_* fields; the source scripts'
agent-only plumbing (artifact IDs, network instrumentation) is dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
🤖 Code Review · Skipped — PR has too many files for review (limit: 1000) |
The doc-search --api-name flag is free-form: valid values are defined by shopify.dev (not the CLI), documented examples are admin/storefront/ hydrogen/functions, and unrecognized values are silently ignored (no error, unlike --api-version). Stop inventing values (customer/partner/ payments/liquid/polaris), tell the agent to omit it when unsure, and fix the illustrative polaris -> app-home. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Model omitting --api-name when the value isn't a documented one, consistent with the clarified guidance. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add unified Shopify umbrella skill
Contributor
Author
|
/snapit |
Contributor
|
🫰✨ Thanks @craigmichaelmartin! Your snapshot has been published to npm. Test the snapshot by installing your package globally: pnpm i -g --@shopify:registry=https://registry.npmjs.org @shopify/cli@0.0.0-snapshot-20260721185120Caution After installing, validate the version by running |
List the full set of shopify.dev doc-search API names in the "search the docs first" step so agents pass only valid --api-name values instead of inventing them: admin, admin-extensions, checkout-ui-extensions, customer-account-ui-extensions, pos-ui-extensions, app-home, storefront, partner, customer, payments-apps, hydrogen, liquid, storefront-web-components, functions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Undo the changes from 80bb226 ("Clarify --api-name is an optional, unvalidated hint") and the enumeration layered on top of it, restoring the prior wording: the inline example list in the doc-search snippet, the simple "scope to a surface / omit to search all" bullet, the Themes --api-name liquid example, the polaris label, and the original gotcha. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
List the full set of shopify.dev doc-search API names in the "search the docs first" step, phrased as a plain enumeration without dismissive framing: admin, admin-extensions, checkout-ui-extensions, customer-account-ui-extensions, pos-ui-extensions, app-home, storefront, partner, customer, payments-apps, hydrogen, liquid, storefront-web-components, functions. Also correct the illustrative snippet's `payments` to `payments-apps`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Differences in type declarationsWe detected differences in the type declarations generated by Typescript for this branch compared to the baseline ('main' branch). Please, review them to ensure they are backward-compatible. Here are some important things to keep in mind:
New type declarationsWe found no new type declarations in this PR Existing type declarationspackages/cli-kit/dist/public/node/metadata.d.ts@@ -34,7 +34,7 @@ export type SensitiveSchema<T> = T extends RuntimeMetadataManager<infer _TPublic
* @returns A container for the metadata.
*/
export declare function createRuntimeMetadataContainer<TPublic extends AnyJson, TSensitive extends AnyJson = Record<string, never>>(defaultPublicMetadata?: Partial<TPublic>): RuntimeMetadataManager<TPublic, TSensitive>;
-type CmdFieldsFromMonorail = PickByPrefix<MonorailEventPublic, 'cmd_all_'> & PickByPrefix<MonorailEventPublic, 'cmd_app_'> & PickByPrefix<MonorailEventPublic, 'cmd_create_app_'> & PickByPrefix<MonorailEventPublic, 'cmd_theme_'> & PickByPrefix<MonorailEventPublic, 'store_'> & PickByPrefix<MonorailEventPublic, 'env_auto_upgrade_'>;
+type CmdFieldsFromMonorail = PickByPrefix<MonorailEventPublic, 'cmd_all_'> & PickByPrefix<MonorailEventPublic, 'cmd_app_'> & PickByPrefix<MonorailEventPublic, 'cmd_create_app_'> & PickByPrefix<MonorailEventPublic, 'cmd_theme_'> & PickByPrefix<MonorailEventPublic, 'cmd_validate_'> & PickByPrefix<MonorailEventPublic, 'store_'> & PickByPrefix<MonorailEventPublic, 'env_auto_upgrade_'>;
declare const coreData: RuntimeMetadataManager<CmdFieldsFromMonorail, {
commandStartOptions: {
startTime: number;
packages/cli-kit/dist/public/node/monorail.d.ts@@ -77,6 +77,11 @@ export interface Schemas {
cmd_app_validate_valid?: Optional<boolean>;
cmd_app_validate_issue_count?: Optional<number>;
cmd_app_validate_file_count?: Optional<number>;
+ cmd_validate_subcommand?: Optional<string>;
+ cmd_validate_result?: Optional<string>;
+ cmd_validate_api?: Optional<string>;
+ cmd_validate_api_version?: Optional<string>;
+ cmd_validate_json?: Optional<boolean>;
cmd_dev_tunnel_type?: Optional<string>;
cmd_dev_tunnel_custom_hash?: Optional<string>;
cmd_dev_urls_updated?: Optional<boolean>;
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a new top-level
shopify validatecommand that ports the four validation scripts fromai-toolkit-source'sshopify-dev-toolspackage into first-class CLI subcommands for offline, deterministic validation:shopify validate theme@shopify/theme-check-*(already shipped by the CLI)shopify validate graphqlgraphql+ bundled introspection JSONshopify validate functionsshopify validate components@shopify/ui-extensionstype defsHow
All four subcommands sit on one shared engine under
packages/cli/src/cli/services/validate/engine/:ValidationResultconst-union,ValidationResponse) and a barrel-decoupled formatter — so the non-component validators never pull in the TypeScript compiler./assetsalike.Behavior preserved from the source tools (flags,
--jsonshape{success, responses, resolvedVersion?}, exit codes, stdin), while adapting to CLI conventions: cli-kit rendering,outputResult+AbortSilentErrorfor--json, telemetry viaaddPublicMetadata(newcmd_validate_*fields). Agent-only plumbing (random-UUID artifact IDs,shopify.devnetwork instrumentation) is dropped for deterministic output. Bad--file/empty stdin now return a structured FAILED payload instead of crashing.Reference data (~13.7 MB compressed: GraphQL schemas + UI-extension
.d.ts.gz) is bundled for fully offline validation and copied intodist/via a recursive esbuild rule.typescriptbecomes a lazily-loaded, esbuild-external runtime dependency (used only byvalidate components).Test plan
nx type-check cli✅ ·nx lint cli✅nx vitest clifor the validate suites: 81/81 pass (real temp dirs, no fs mocking)nx bundle cli✅ — bundled binary smoke-tested for all four subcommands:validate graphql --api admin --code 'query { shop { name } }'→success, exit 0validate graphqlwith an invalid field →failed, exit 1 (Cannot query field "nope" on type "Shop")validate functions --api functions_discount(valid input query) →successvalidate theme(codeblock) →successvalidate components --api polaris-app-home --code '<s-text>hello</s-text>' --language html→success,validatedComponents: ["s-text"]oclif.manifest.json,README.md, and dev-docs; changeset added.Notes / reviewer callouts
typescript(lazy, esbuild-external) forvalidate components.packages/cli/assets/validate/.gitignorere-includes the vendored type packages'dist/folders (otherwise the repo-widedistignore would drop ~320.d.ts.gzreference files).🤖 Generated with Claude Code