Skip to content

Add 'excerpter' package to the tools repository#2411

Open
kevmoo wants to merge 11 commits into
mainfrom
excerpter
Open

Add 'excerpter' package to the tools repository#2411
kevmoo wants to merge 11 commits into
mainfrom
excerpter

Conversation

@kevmoo
Copy link
Copy Markdown
Member

@kevmoo kevmoo commented May 20, 2026

Migrate the excerpter package from dart-lang/labs to dart-lang/tools.

Core Package & Setup:

  • Ported all core libraries, executable entry points, and tests to pkgs/excerpter.
  • Adapted constraints to target Dart SDK ^3.12.0 and replaced the site-specific analysis_defaults with package:dart_flutter_team_lints.
  • Structured a clean library programmatic usage example in example/main.dart.
  • Exposed code ownership in .github/CODEOWNERS.
  • Set up GitHub Actions CI workflow in .github/workflows/excerpter.yaml.
  • Set up labeler patterns in .github/labeler.yml.
  • Registered the excerpter package in the root README.md.

Migrate the excerpter package from dart-lang/labs to dart-lang/tools.

Core Package & Setup:
- Ported all core libraries, executable entry points, and tests to `pkgs/excerpter`.
- Adapted constraints to target Dart SDK `^3.9.0` and replaced the site-specific `analysis_defaults` with `package:dart_flutter_team_lints`.
- Structured a clean library programmatic usage example in `example/main.dart`.
- Exposed code ownership in `.github/CODEOWNERS`.
- Set up GitHub Actions CI workflow in `.github/workflows/excerpter.yaml`.
- Set up labeler patterns in `.github/labeler.yml`.
- Registered the `excerpter` package in the root `README.md`.

Core Bug Fix & Enhancements:
- Fixed a `RangeError` crash inside the code block parser loop in `inject.dart` when dealing with unclosed blocks at the end of a file.
- Implemented native tracking of static code blocks to ignore illustrative `<?code-excerpt` snippets inside fenced Markdown regions, avoiding visual backslash prefix workarounds.
- Formatted all files to adhere strictly to the 80-column limit.

Testing & Robustness:
- Added dedicated unit tests for exceptional extractor parser failures in `test/extractor_test.dart` and `test/injection_test.dart`.
- Migrated temporary directory setups to use `package:test_descriptor` for platform-agnostic, isolated sandbox tests.
- Added an automated validation test in `test/cli_test.dart` to guarantee the `README.md` options table stays in perfect sync with the CLI `ArgParser` definition.
- Achieved 87.6% core statement coverage (including 100% coverage on `extract.dart` and `update.dart`).
@kevmoo kevmoo requested a review from a team as a code owner May 20, 2026 03:51
@github-actions github-actions Bot added the type-infra A repository infrastructure change or enhancement label May 20, 2026
@github-actions
Copy link
Copy Markdown

Package publishing

Package Version Status Publish tag (post-merge)
package:bazel_worker 1.1.5 already published at pub.dev
package:benchmark_harness 2.4.0 already published at pub.dev
package:boolean_selector 2.1.2 already published at pub.dev
package:browser_launcher 1.2.0-wip WIP (no publish necessary)
package:cli_config 0.2.1-wip WIP (no publish necessary)
package:cli_util 0.5.1 already published at pub.dev
package:clock 1.1.3-wip WIP (no publish necessary)
package:code_builder 4.12.0-wip WIP (no publish necessary)
package:coverage 1.15.0 already published at pub.dev
package:csslib 1.0.2 already published at pub.dev
package:excerpter 0.1.0-wip WIP (no publish necessary)
package:extension_discovery 2.1.0 already published at pub.dev
package:file 7.0.2-wip WIP (no publish necessary)
package:file_testing 3.1.0-wip WIP (no publish necessary)
package:glob 2.1.3 already published at pub.dev
package:graphs 2.4.0-wip WIP (no publish necessary)
package:html 0.15.7-wip WIP (no publish necessary)
package:io 1.1.0-wip WIP (no publish necessary)
package:json_rpc_2 4.1.0 already published at pub.dev
package:markdown 7.4.0 ready to publish markdown-v7.4.0
package:mime 2.1.0-wip WIP (no publish necessary)
package:oauth2 2.0.5 already published at pub.dev
package:package_config 2.3.0-wip WIP (no publish necessary)
package:pool 1.5.3-wip WIP (no publish necessary)
package:process 5.0.5 (error) pubspec version (5.0.5) and changelog (5.0.6-wip) don't agree
package:pub_semver 2.2.0 already published at pub.dev
package:pubspec_parse 1.6.0-wip WIP (no publish necessary)
package:source_map_stack_trace 2.1.3-wip WIP (no publish necessary)
package:source_maps 0.10.14-wip WIP (no publish necessary)
package:source_span 1.10.2 already published at pub.dev
package:sse 4.2.0 already published at pub.dev
package:stack_trace 1.12.2-wip (error) pubspec version (1.12.2-wip) and changelog (1.12.2-dev) don't agree
package:stream_channel 2.1.4 already published at pub.dev
package:stream_transform 2.1.2-wip WIP (no publish necessary)
package:string_scanner 1.4.2-wip WIP (no publish necessary)
package:term_glyph 1.2.3-wip WIP (no publish necessary)
package:test_reflective_loader 0.6.0 ready to publish test_reflective_loader-v0.6.0
package:timing 1.0.2 already published at pub.dev
package:unified_analytics 8.0.15 ready to publish unified_analytics-v8.0.15
package:watcher 1.2.2-wip WIP (no publish necessary)
package:yaml 3.1.4-wip WIP (no publish necessary)
package:yaml_edit 2.2.4 already published at pub.dev

Documentation at https://github.com/dart-lang/ecosystem/wiki/Publishing-automation.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 20, 2026

PR Health

License Headers ✔️
// Copyright (c) 2026, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.

Files
no missing headers

All source files should start with a license header.

Unrelated files missing license headers
Files
pkgs/bazel_worker/benchmark/benchmark.dart
pkgs/coverage/lib/src/coverage_options.dart
pkgs/html/example/main.dart
pkgs/pubspec_parse/test/git_uri_test.dart
pkgs/watcher/test/custom_watcher_factory_test.dart

This check can be disabled by tagging the PR with skip-license-check.

Breaking changes ⚠️
Package Change Current Version New Version Needed Version Looking good?
excerpter Breaking 0.0.0 0.0.0 0.0.0
Report was not created for package:excerpter 0.1.0-wip (dir=/home/runner/work/tools/tools/current_repo/pkgs/excerpter). Error: Error: Exception: Package not available (could not find package excerpter at https://pub.dev).
⚠️

This check can be disabled by tagging the PR with skip-breaking-check.

Coverage ⚠️
File Coverage
pkgs/excerpter/bin/excerpter.dart 💔 Not covered
pkgs/excerpter/example/main.dart 💔 Not covered
pkgs/excerpter/lib/excerpter.dart 💔 Not covered
pkgs/excerpter/lib/src/extract.dart 💚 98 %
pkgs/excerpter/lib/src/inject.dart 💚 75 %
pkgs/excerpter/lib/src/transform.dart 💚 98 %
pkgs/excerpter/lib/src/update.dart 💚 100 %
pkgs/excerpter/test_data/example/dartdoc.dart 💔 Not covered
pkgs/excerpter/test_data/example/plaster.dart 💔 Not covered
pkgs/excerpter/test_data/example/simple_main.dart 💔 Not covered
pkgs/excerpter/test_data/example/simple_region.dart 💔 Not covered
pkgs/excerpter/test_data/example/transforms.dart 💔 Not covered

This check for test coverage is informational (issues shown here will not fail the PR).

This check can be disabled by tagging the PR with skip-coverage-check.

API leaks ✔️

The following packages contain symbols visible in the public API, but not exported by the library. Export these symbols or remove them from your publicly visible API.

Package Leaked API symbol Leaking sources

This check can be disabled by tagging the PR with skip-leaking-check.

Unused Dependencies ✔️
Package Status
excerpter ✔️ All dependencies utilized correctly.

For details on how to fix these, see dependency_validator.

This check can be disabled by tagging the PR with skip-unused-dependencies-check.

Changelog Entry ✔️
Package Changed Files

Changes to files need to be accounted for in their respective changelogs.

This check can be disabled by tagging the PR with skip-changelog-check.

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the excerpter package, a tool for synchronizing code excerpts in Markdown files with source code regions. It includes a CLI, a programmatic API, and various text transformations. Feedback focuses on improving the robustness of the instruction parser (specifically handling escaped quotes and delimiters), ensuring consistency between the documentation and regex for region names, and optimizing performance by pre-compiling regexes and sharing the extraction cache across multiple files.

Comment thread pkgs/excerpter/lib/src/inject.dart
Comment thread pkgs/excerpter/README.md Outdated
Comment thread pkgs/excerpter/lib/src/extract.dart
Comment thread pkgs/excerpter/lib/src/inject.dart Outdated
Comment thread pkgs/excerpter/lib/src/inject.dart
Comment thread pkgs/excerpter/lib/src/update.dart
@kevmoo
Copy link
Copy Markdown
Member Author

kevmoo commented May 20, 2026

/gemini review

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the excerpter package to the monorepo, a tool designed to synchronize code excerpts in Markdown files with regions defined in source code. The reviewer identified several areas for improvement, including a potential RangeError in inject.dart when handling short lines and a bug in the shared indentation logic that fails when excerpts contain empty lines. Additionally, the feedback suggests using the path package for more robust file extension detection and updating the README.md to reflect the package's new repository location and bug tracker links.

Comment thread pkgs/excerpter/lib/src/inject.dart
Comment thread pkgs/excerpter/lib/src/inject.dart
Comment thread pkgs/excerpter/README.md Outdated
Comment thread pkgs/excerpter/README.md Outdated
Comment thread pkgs/excerpter/lib/src/extract.dart Outdated
@kevmoo
Copy link
Copy Markdown
Member Author

kevmoo commented May 20, 2026

/gemini review

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new package, excerpter, designed to update code excerpts in Markdown documentation from source files. The changes include the package structure, CLI implementation, and necessary configuration files. The review identified a discrepancy between the SDK version in the PR description and the pubspec.yaml, suggested simplifying argument parsing and error handling in the CLI, and pointed out an unnecessary await in pkgs/markdown/tool/update_shared.dart that conflicts with the unnecessary_await_in_return lint rule.

Comment thread pkgs/excerpter/pubspec.yaml
Comment thread pkgs/excerpter/bin/excerpter.dart Outdated
Comment thread pkgs/excerpter/bin/excerpter.dart Outdated
Comment thread pkgs/markdown/tool/update_shared.dart Outdated
@kevmoo
Copy link
Copy Markdown
Member Author

kevmoo commented May 20, 2026

/gemini review

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the excerpter package, a tool for synchronizing code excerpts in Markdown documentation with source files. It includes a CLI, extraction and injection logic, and various text transformations. Review feedback identifies a potential path traversal vulnerability in path joining, suggests normalizing paths in the region cache to prevent redundant processing, and recommends replacing a magic number with a named constant.

Comment thread pkgs/excerpter/lib/src/inject.dart Outdated
Comment thread pkgs/excerpter/lib/src/extract.dart
Comment thread pkgs/excerpter/lib/src/extract.dart Outdated
Comment thread pkgs/excerpter/README.md

To include lines within a code region,
add a Dart line comment before the first line in the region with
`#docregion <name>` where `<name>` is your desired name for the region.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kevmoo please look at dart-lang/dartdoc#4144

Can talk to others and see if we can agree on something.

IMO, it's preferable if this is as simple as possible.

If we want to have substitution, what is the motivation? And are we sure the benefits outweigh the costs of duplicating a few examples.

I can see how a reference using skip=5 could be hard to maintain, how will I when editing the code file that contains the doc-region know that it's referenced somewhere with skip=5 and that this has to be meaningful.

I guess you'll be able to see it in the PR diff (if you read all the diff, but not if you skim it). But for something like {$example path/to/example/code.dart#myregion} a skip=5 option would be dangerous, because when editing path/to/example/code.dart I can see #region myregion but I can't really know that one place the region is used with skip-5 and in another it's not. And with dartdoc API doc changes won't be visible in the PR.

The same concern applies with substituions, etc. this is my main concern: too many features makes updating code examples dangerous, because it hard to know how the docregion is being referenced (and what substitutions it is subject to).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@parlough do you have some context on all the options here?

Do we need them? and is it really wise to have so many options.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

package:excerpter type-infra A repository infrastructure change or enhancement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants