refactor(es/lexer): token eof #10880

bvanjoi · 2025-07-17T12:43:44Z

No description provided.

changeset-bot · 2025-07-17T12:43:52Z

⚠️ No Changeset found

Latest commit: f32db4a

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

codspeed-hq · 2025-07-17T13:21:05Z

CodSpeed Performance Report

Merging #10880 will improve performances by 3.67%

_{Comparing bvanjoi:parse-error (f32db4a) with dev/rust (fc680c2)¹}

Summary

⚡ 13 improvements
✅ 127 untouched benchmarks

Benchmarks breakdown

	Benchmark	`BASE`	`HEAD`	Change
⚡	`es/codegen/with-parser/colors`	159.6 µs	155.4 µs	+2.64%
⚡	`es/parser/angular`	20.4 ms	19.9 ms	+2.16%
⚡	`es/parser/backbone`	3.2 ms	3.1 ms	+2.3%
⚡	`es/parser/cal-com`	64.9 ms	63.1 ms	+2.91%
⚡	`es/parser/colors`	90.8 µs	88.7 µs	+2.37%
⚡	`es/parser/jquery`	16.5 ms	16.1 ms	+2.31%
⚡	`es/parser/jquery mobile`	25.6 ms	25.1 ms	+2.16%
⚡	`es/parser/mootools`	12.9 ms	12.6 ms	+2.56%
⚡	`es/parser/three`	78.5 ms	76.7 ms	+2.37%
⚡	`es/parser/typescript`	435 ms	425.7 ms	+2.17%
⚡	`es/parser/underscore`	2.8 ms	2.7 ms	+2.37%
⚡	`es/parser/yui`	12.6 ms	12.3 ms	+2.3%
⚡	`typescript/fast-strip`	532.3 µs	513.4 µs	+3.67%

No successful run was found on dev/rust (1c6544e) during the generation of this report, so fc8dc84 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

magic-akari · 2025-07-17T15:07:27Z

This appears to be an unexpected closure triggered by the deletion of the target branch.

kdy1 · 2025-07-17T15:59:12Z

Nice catch, I was wondering where did the PR go 🤣

kdy1 · 2025-07-18T11:34:20Z

I think this is a good improvement. What matters is the performance of the parser.

It seems like this PR moves some work of the parser to the lexer, but the total work done by the parser is reduced.

socket-security · 2025-07-18T12:37:21Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Quality	Maintenance
	@babel/traverse@7.22.1
	@types/terser@3.12.0
	@swc/counter@0.1.3
	node-releases@2.0.13
	@types/parse-json@4.0.0
	is-arrayish@0.2.1
	@swc/plugin-jest@1.5.117
	tmpl@1.0.5
	@babel/plugin-transform-react-jsx-development@7.18.6
	has@1.0.3
	jest-regex-util@25.2.6 ⏵ 29.6.3			^-1
	jest-get-type@25.2.6 ⏵ 29.6.3		⁺¹
	@jest/globals@25.5.2 ⏵ 29.7.0		⁺¹
	@babel/plugin-transform-unicode-regex@7.18.6
	escape-string-regexp@2.0.0 ⏵ 1.0.5		^-5
	@jest/source-map@25.5.0 ⏵ 29.6.3			^-1
	@babel/plugin-transform-unicode-sets-regex@7.22.3
	@babel/plugin-transform-exponentiation-operator@7.18.6
	@babel/plugin-transform-reserved-words@7.18.6
	@babel/plugin-transform-private-methods@7.22.3
	core-util-is@1.0.2 ⏵ 1.0.3		^-1
	@babel/plugin-transform-sticky-regex@7.18.6
	collect-v8-coverage@1.0.1
	@babel/plugin-transform-named-capturing-groups-regex@7.22.3
	jest-resolve-dependencies@25.5.4 ⏵ 29.7.0	⁺¹	⁺¹
	jest-environment-node@25.5.0 ⏵ 29.7.0	⁺¹	⁺¹
	@babel/plugin-transform-property-literals@7.18.6
	@babel/plugin-transform-optional-catch-binding@7.22.3
	@babel/plugin-transform-unicode-property-regex@7.22.3
	@jest/schemas@29.6.3
	@jest/expect@29.7.0
	@babel/plugin-transform-member-expression-literals@7.18.6
	strip-final-newline@2.0.0
See 413 more rows in the dashboard

View full report

socket-security · 2025-07-18T12:37:22Z

Warning

Review the following alerts detected in dependencies.

According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.

Action	Severity	Alert (click for details)
Warn		`@babel/[email protected]` has a Critical CVE. CVE: GHSA-67hx-6x53-jw92 Babel vulnerable to arbitrary code execution when compiling specifically crafted malicious code (CRITICAL) Affected versions: < 7.23.2; >= 8.0.0-alpha.0 < 8.0.0-alpha.4 Patched version: 7.23.2 From: crates/swc_ecma_minifier/yarn.lock → `npm/@babel/[email protected]` ℹ Read more on: This package \| This alert \| What is a critical CVE? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at `[email protected]`. Suggestion: Remove or replace dependencies that include known critical CVEs. Consumers can use dependency overrides or npm audit fix --force to remove vulnerable dependencies. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment `@SocketSecurity ignore npm/@babel/[email protected]`. You can also ignore all packages with `@SocketSecurity ignore-all`. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

View full report

bvanjoi · 2025-07-19T08:00:31Z

crates/swc_ecma_transforms_base/tests/ts-resolver/ts_resolver_nested_enum/output.ts

@@ -1,11 +1,9 @@
 enum Foo__2 {
-    name__0,
-    string__0
+    name__0


I think this change is acceptable because the input(ts_resolver_nested_enum) contains invalid syntax.

Copilot

Pull Request Overview

This pull request refactors the ECMAScript lexer to introduce an explicit EOF (End of File) token rather than using None/Option to represent the end of input. This is a significant architectural change that makes EOF handling more explicit and consistent throughout the parser.

Introduces a new Token::Eof variant in both swc_ecma_parser and swc_ecma_lexer crates
Refactors token handling to use &Token instead of Option<&Token> for current token access
Updates parser initialization to consume the initial EOF token and ensures proper EOF handling at the end of parsing

Reviewed Changes

Copilot reviewed 73 out of 83 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
crates/swc_ecma_parser/src/lexer/token.rs	Adds `Eof` token variant and implements necessary token factory methods
crates/swc_ecma_parser/src/parser/input.rs	Major refactor to use `TokenAndSpan` directly instead of `Option<TokenAndSpan>`
crates/swc_ecma_parser/src/parser/mod.rs	Updates parser initialization and ensures EOF consumption after parsing
crates/swc_ecma_lexer/src/token.rs	Adds `Eof` token variant to lexer token enum
crates/swc_ecma_lexer/src/lexer/table.rs	Updates byte handlers to return `Token` instead of `Option<Token>`
Multiple parser files	Updates throughout to handle explicit EOF token instead of None checks

Copilot · 2025-07-19T13:10:07Z

crates/swc_ecma_lexer/src/token.rs

@@ -956,6 +967,7 @@ impl Token {
            Self::JSXTagEnd => TokenKind::JSXTagEnd,
            Self::Shebang(..) => TokenKind::Shebang,
            Self::Error(..) => TokenKind::Error,
+            Self::Eof => TokenKind::Error,


EOF token should not map to TokenKind::Error. This creates semantic confusion as EOF is a valid end-of-input state, not an error condition. Consider creating a dedicated TokenKind::Eof or mapping to a more appropriate kind.

Suggested change

Self::Eof => TokenKind::Error,

Self::Eof => TokenKind::Eof,

bvanjoi requested review from a team as code owners July 17, 2025 12:43

bvanjoi marked this pull request as draft July 17, 2025 12:43

kdy1 added this to the Planned milestone Jul 17, 2025

kdy1 deleted the branch swc-project:dev/rust July 17, 2025 14:57

kdy1 closed this Jul 17, 2025

kdy1 reopened this Jul 17, 2025

bvanjoi force-pushed the parse-error branch from cd8d70f to 6081335 Compare July 17, 2025 16:11

CPunisher mentioned this pull request Jul 18, 2025

perf(es/lexer): Remove option for handler #10881

Draft

bvanjoi force-pushed the parse-error branch from 6081335 to c8b7188 Compare July 18, 2025 07:03

bvanjoi force-pushed the parse-error branch from c8b7188 to b57d350 Compare July 18, 2025 12:36

bvanjoi force-pushed the parse-error branch from b57d350 to 1b87cde Compare July 18, 2025 16:30

bvanjoi marked this pull request as ready for review July 18, 2025 16:45

refactor(es/lexer): token eof

f32db4a

bvanjoi force-pushed the parse-error branch from 1b87cde to f32db4a Compare July 19, 2025 07:59

bvanjoi commented Jul 19, 2025

View reviewed changes

kdy1 requested a review from Copilot July 19, 2025 13:09

Copilot AI reviewed Jul 19, 2025

View reviewed changes

kdy1 approved these changes Jul 19, 2025

View reviewed changes

kdy1 merged commit eb5819a into swc-project:dev/rust Jul 19, 2025
172 of 173 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

refactor(es/lexer): token eof #10880

refactor(es/lexer): token eof #10880

bvanjoi commented Jul 17, 2025

Uh oh!

changeset-bot bot commented Jul 17, 2025 •

edited

Loading

Uh oh!

codspeed-hq bot commented Jul 17, 2025 •

edited

Loading

Uh oh!

magic-akari commented Jul 17, 2025

Uh oh!

kdy1 commented Jul 17, 2025

Uh oh!

kdy1 commented Jul 18, 2025

Uh oh!

socket-security bot commented Jul 18, 2025 •

edited

Loading

Uh oh!

socket-security bot commented Jul 18, 2025 •

edited

Loading

Uh oh!

bvanjoi Jul 19, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jul 19, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

refactor(es/lexer): token eof #10880

refactor(es/lexer): token eof #10880

Conversation

bvanjoi commented Jul 17, 2025

Uh oh!

changeset-bot bot commented Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

codspeed-hq bot commented Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CodSpeed Performance Report

Merging #10880 will improve performances by 3.67%

Summary

Benchmarks breakdown

Footnotes

Uh oh!

magic-akari commented Jul 17, 2025

Uh oh!

kdy1 commented Jul 17, 2025

Uh oh!

kdy1 commented Jul 18, 2025

Uh oh!

socket-security bot commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

socket-security bot commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bvanjoi Jul 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Jul 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

changeset-bot bot commented Jul 17, 2025 •

edited

Loading

codspeed-hq bot commented Jul 17, 2025 •

edited

Loading

socket-security bot commented Jul 18, 2025 •

edited

Loading

socket-security bot commented Jul 18, 2025 •

edited

Loading