Skip to content

fix highlighter bugs - #4022

Merged
JoeKar merged 4 commits into
micro-editor:masterfrom
redsti-github:fix/highlight
Nov 17, 2026
Merged

JoeKar merged 4 commits into
micro-editor:masterfrom
redsti-github:fix/highlight

Conversation

@redsti-github

Copy link
Copy Markdown
Contributor

(new PR since i accidentally closed the old one (#4020), sorry)

@Neko-Box-Coder

Neko-Box-Coder commented Jul 31, 2026 •

Copy link
Copy Markdown
Contributor

@JoeKar

Hey, I think it will be great to get this reviewed and merged because I am seeing this problem manifests on a regular basis, and also considering it seems like #3127 is not going to be completed anytime soon.

@JoeKar

JoeKar commented Aug 2, 2026

Copy link
Copy Markdown
Member

If it doesn't break anything else then yes.

@Neko-Box-Coder

Copy link
Copy Markdown
Contributor

@JoeKar

Okay, I will try daily driving this and see how it goes.

Sorry to nag but would be great if you can take a look at #4028 when you are free as I have daily driven/tested it for a month and seems good to me.

@Londopy

Londopy commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

I hit this same bug from a different direction (a #include region in #4236 — short header names made it show up constantly) and opened a duplicate fix before finding this PR. Closed mine; this one is strictly more complete.

Since the open question here was "if it doesn't break anything else", and pkg/highlight currently has no tests, here's a regression test that might help. It puts a region at every start offset on the line and checks the rule inside it still applies — it fails at offset 3 on master and passes on this branch. I ran the full suite on top of this branch and everything is green.

Verified on 847ac7e:

$ go test ./pkg/highlight/
ok      github.com/micro-editor/micro/v2/pkg/highlight  0.262s

$ go test ./...
ok      github.com/micro-editor/micro/v2/cmd/micro
ok      github.com/micro-editor/micro/v2/internal/buffer
ok      github.com/micro-editor/micro/v2/internal/config
ok      github.com/micro-editor/micro/v2/internal/util
ok      github.com/micro-editor/micro/v2/internal/views
ok      github.com/micro-editor/micro/v2/pkg/highlight
ok      github.com/micro-editor/micro/v2/runtime
pkg/highlight/highlighter_test.go
package highlight

import (
	"strings"
	"testing"
)

const regionSyntax = `filetype: test

detect:
    filename: "\\.test$"

rules:
    - comment:
        start: "//"
        end: "$"
        rules:
            - todo: "TODO"
`

func testHighlighter(t *testing.T, syntax string) *Highlighter {
	t.Helper()
	f, err := ParseFile([]byte(syntax))
	if err != nil {
		t.Fatalf("ParseFile: %v", err)
	}
	def, err := ParseDef(f, &Header{})
	if err != nil {
		t.Fatalf("ParseDef: %v", err)
	}
	return NewHighlighter(def)
}

// The rules inside a region must be applied no matter where on the line the
// region begins. The end match is located in the region's own slice of the
// line, so its index must not be compared against the region's absolute start
// offset: when the two happened to be equal, every rule inside the region was
// skipped and the contents were left with the region's own group.
func TestRegionRulesAppliedAtAnyStartOffset(t *testing.T) {
	for prefix := 0; prefix < 24; prefix++ {
		line := strings.Repeat("x", prefix) + "// TODO"

		h := testHighlighter(t, regionSyntax)

		todo, ok := Groups["todo"]
		if !ok {
			t.Fatal("todo group not defined")
		}

		matches := h.HighlightString(line)

		found := false
		for _, g := range matches[0] {
			if g == todo {
				found = true
				break
			}
		}
		if !found {
			t.Errorf("prefix %d: %q: todo inside the comment was not highlighted", prefix, line)
		}
	}
}

Happy to open it as a separate PR against master once this lands, or you're welcome to just take it into this branch — whichever is easier.

@JoeKar JoeKar left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Performs better in various use cases, where the actual implementation fails.
👍

@Londopy Londopy mentioned this pull request Sep 23, 2026
@Neko-Box-Coder

Copy link
Copy Markdown
Contributor

Totally forgot about this but yeah, I haven't encountered any problem daily driving this PR.

@JoeKar

JoeKar commented Sep 26, 2026

Copy link
Copy Markdown
Member

@dmaluka:
I suggest to merge it, because it seems to improve the current implementation. No regression was found so far.
What do you think?

Regarding #3127 I'm on it.

@dmaluka

dmaluka commented Sep 26, 2026

Copy link
Copy Markdown
Collaborator

It would be nice if the commit message explained in a bit more detail what exact bugs this fixes and how exactly it fixes them (given that the highlighter code is by no means easy to understand).

i.e.:

patterns within regions no longer fail on certain line lengths and columns (#4018)

How exactly does it make them no longer fail?

region applies patterns before next nested region

What exactly is the issue and what exactly is the fix? (and BTW, if this is a separate issue, why is it in the same commit?)

Comment thread pkg/highlight/highlighter.go Resolved
@redsti-github

Copy link
Copy Markdown
Contributor Author

I've updated the commit message to more accurately describe the changes, and added @Londopy 's test and one more.

@dmaluka

dmaluka commented Sep 28, 2026

Copy link
Copy Markdown
Collaborator

Thanks. Sounds like there are three separate fixes? Any chance to split them correspondingly into 3 commits, so that it is clearly seen which change fixes what? (The test also could be in its own commit.)

Also I see it still only describes what problems it fixes, not how it fixes them. What exactly was the problem with start == endLoc[0], and why exactly replacing it with endLoc[0] == 0 is correct? (IIUC endLoc[0] == 0 checks if the found occurrence of the region's end regex is at the beginning of a line? why???) And so on.

...Also please fix the gofmt issues in highlighter_test.go causing the CI checks to fail. (@JoeKar good news: your #4175 proves to work in the wild.)

@Londopy

Londopy commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

On the start == endLoc[0] part, since I went through that code for #4240:

line in highlightRegion isn't the whole line, it's the part from column start onward (callers pass e.g. sliceStart(line, firstLoc[1]) together with start+firstLoc[1]). findIndex returns positions relative to that slice, which is why the other uses add start (highlights[start+loc[0]] etc.).

So endLoc[0] == 0 means the region's end is found right at start: nothing is left inside the region, so there's nothing to search. (Skipping is only a shortcut there, since no nested region or pattern could start before index 0 anyway.)

start == endLoc[0] compares that slice-relative index with the absolute column. It's the same test when start is 0 (a region continued from the previous line), but anywhere else it's true by coincidence, whenever the end happens to be start characters into the slice. Nested regions and patterns then get skipped even though there's still text inside the region. On master, in C:

  • ab;// TODO: the comment's contents start at column 5 and its end ($) is 5 characters into them, so TODO isn't highlighted
  • a;// TODO: 4 vs 5, so it is

@redsti-github feel free to use any of this in the commit message.

@jv-k

jv-k commented Oct 2, 2026

Copy link
Copy Markdown

This comparison can help with the question "if it doesn't break anything else".

For each built-in syntax file, I highlighted each matching file in the micro repository with HighlightString: 325 files and 39,197 lines. I did this with master and with each commit of this PR, then compared the results line by line.

Commit Changed lines
d0c5271d patterns within regions no longer fail on specific line lengths and columns 32
e3c31f2e nested regions do not skip formatting parent region 82
bd8f77e6 regions patterns only match inside the region 155

At the PR head, 105 of the 155 lines are better:

  • 73 lines are Go character literals and format strings. On master, the error: "..+" and %. patterns in go.yaml match past the closing quote. Thus 'y' shows as an error, and the % in "%" shows as a format verb.
  • 31 lines are strings that lost their escape colours (syntax highlighting fails on certain columns and line widths #4018).
  • 1 line is an XML attribute value that now shows as a string.

20 lines have no visible change.

30 lines are worse. All of them are XML lines where more tags follow a self-closing tag. An example is line 25 of assets/micro-logo-drop.svg:

           rdf:resource="http://purl.org/dc/dcmitype/StillImage" /></cc:Work></rdf:RDF></metadata><defs

On master, the text after /> has the symbol.tag colour. From e3c31f2e, it has the identifier colour. In xml.yaml, the identifier region starts at a space and ends at =. After />, no = follows on the line, thus the region continues to the end of the line.

@redsti-github

redsti-github commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Thank you for giving this a good test!

30 lines are worse. All of them are XML lines where more tags follow a self-closing tag.

In my opinion, this is definitely an issue with xml.yaml, rather than this change. The formatting is now "correct" in that it follows xml.yaml properly, but it's not "correct" in that it does "what we want".

I'd propose to change xml.yaml lines 22-25 from

            - identifier:
                start: " "
                end: "="
                rules: []

to

            - identifier: " [a-zA-Z0-9:_-]+=?"

which appears to work correctly. (Though this change should probably be a seperate PR.)

@JoeKar

JoeKar commented Oct 19, 2026

Copy link
Copy Markdown
Member

I'd propose to change xml.yaml lines 22-25 from

            - identifier:
                start: " "
                end: "="
                rules: []

to

            - identifier: " [a-zA-Z0-9:_-]+=?"

It is more trivial, but it doesn't highlight this valid XML any longer:

		<marker
				 id
				=
				"arrow">
			[...]
		</marker>

A corrected version of the highlighter should do this.
If we change it, then we should generalize the start to \s to allow the valid newlines and TABs.

@redsti-github

Copy link
Copy Markdown
Contributor Author

Ah, i forgot attributes can span multiple lines.
In that case

            - identifier:
                start: "\\s[:A-Z_a-z]"
                end: "="
                rules: []

should work. (Though xml spec allows attributes to start with certain unicode characters. I'm not sure how to handle that.)

Either way, this bug is still a problem with xml.yaml and not the highlighter.

@dmaluka

dmaluka commented Oct 30, 2026

Copy link
Copy Markdown
Collaborator

Re commit messages again:

f.e. "// TODO <...>" will actually format "TODO" if "<...>" is a nested region.

Not sure what this means. What exactly is the issue and how to reproduce it?
What was highlighted incorrectly: // TODO <...> (without double quotes) or "// TODO <...>" (within double quotes), and with which filetype? AFAICS with filetype=c both are highlighted correctly without this PR, just like with this PR.

Also, I assume by "format" you mean "highlight"? Or perhaps something else?

instead of from the region start to end of line ("$" now also matches end of region or start of nested region)

Again, could you give an example of what exactly was highlighted incorrectly?

@redsti-github

Copy link
Copy Markdown
Contributor Author

What exactly is the issue and how to reproduce it?

If there is a nested region inside another nested region (f.e. a text string within a html tag), then everything between the start of the outer region, to the start of the inner region just doesn't get highlighted.

Look at this: <a style="abcd" size=5></a> (not valid html, i know), here "style" should get the identifer colour, but doesn't.

Also check the added highlighter tests, the second one does this check (with a different example than this html one)

Also, I assume by "format" you mean "highlight"? Or perhaps something else?

Yes.

instead of from the region start to end of line ("$" now also matches end of region or start of nested region)

Again, could you give an example of what exactly was highlighted incorrectly?

Here, there is nothing that is highlighted incorrectly, but it's mostly just a small optimisation. Currently, regions will attempt to highlight everything from the start of the region, to the end of the line, and then throw away everything that is past the end of the region.
This commit makes it so it only scans within the region, with the side effect that $ now matches to the end of the region. (In my opinion this isn't an issue, since ^ already matches the start of the region so we get some extra symmetry between the two, and i don't think this change breaks any of the current syntax rules.)

@dmaluka

dmaluka commented Oct 31, 2026

Copy link
Copy Markdown
Collaborator

If there is a nested region inside another nested region (f.e. a text string within a html tag), then everything between the start of the outer region, to the start of the inner region just doesn't get highlighted.

Look at this: <a style="abcd" size=5></a> (not valid html, i know), here "style" should get the identifer colour, but doesn't.

I see. Could you add this explanation to the commit message? (In particular, please add e.g. this <a style="abcd" size=5></a> example, instead of that // TODO <...>. Now I do see that // TODO <...> refers to the test syntax used in TestRegionRulesApplyWithNestedRegion(), but from the commit message that is not clear at all, so it is only confusing.)

...Also, why didn't it get highlighted? The commit message says "fixed by recursing to nested region only after formatting the current one" but why does the order matter?

IOW, what exactly is the root cause?

Also check the added highlighter tests, the second one does this check (with a different example than this html one)

Then makes sense to add that second test in the same commit as this fix (and the first test, correspondingly, in the same commit as the first fix), instead adding both tests in one separate commit? So that it is clear which test is related to which fix.

Here, there is nothing that is highlighted incorrectly, but it's mostly just a small optimisation. Currently, regions will attempt to highlight everything from the start of the region, to the end of the line, and then throw away everything that is past the end of the region.
This commit makes it so it only scans within the region, with the side effect that $ now matches to the end of the region. (In my opinion this isn't an issue, since ^ already matches the start of the region so we get some extra symmetry between the two, and i don't think this change breaks any of the current syntax rules.)

Could you add this info to the commit message?

if !statesOnly {
highlights[start+firstLoc[0]] = firstRegion.limitGroup
}
h.highlightEmptyRegion(highlights, start+firstLoc[1], canMatchEnd, lineNum, sliceStart(line, firstLoc[1]), statesOnly)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see you removed this highlightEmptyRegion() call. Is it because you considered it redundant and pointless (even independently of this fix)?

It seems indeed redundant and pointless: it highlights the same exact range as the highlightRegion() call right below it? So, we first highlight it as an "empty" region i.e. pretending that firstRegion is nil, and then we overwrite this bogus highlighting with the correct highlighting in the below highlightRegion()? What's going on?

Actually it is a question to @JoeKar who added this highlightEmptyRegion() in 7e8ff8c#diff-16d0164b7fcc4350a9a7a5827b930e8de29487a35a3afa8d9e165b8e2af0f13aR161. What was the reason for it? And why was it correct to remove h.highlightRegion(highlights, start, false, lineNum, sliceEnd(line, firstLoc[0]), curRegion, statesOnly), replacing it with this highlightEmptyRegion()?

...Hmm, isn't this whole fix basically a revert of 7e8ff8c? @JoeKar what do you say?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I remember correctly, I introduced this back then to highlight potential current patterns as long as no further nested regions are available.
But to be honest: As I mentioned in another issue, I gave up trying to understand this highlighting logic and discarded it in favor of #3127 (which I more or less exchanged locally too).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAICS 7e8ff8c moved highlighting current region after recursing into nested region, while this fix moves it back before recursing into nested region. So, does this fix break whatever 7e8ff8c was fixing, or not? Can we check that? (Do we remember how to check that?)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, 7e8ff8c just moved the "highlight the part that is past the end of the region and return" part to after "highlight the current region".

But it still has "highlight nested region and return" (lines 177-184) before "highlight the current region" (lines 186-205), which this change swaps.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I see.

Still, what about my earlier question (I'm just curious): did you remove this highlightEmptyRegion() call because you considered it unneeded regardless of this change, or is this removal specifically related to this change?

BTW if the former, it would be probably better to do this removal in a separate commit before this one (for easier review, to not mix two different changes in the same commit)?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed it because it looked either unnecesarry, or plain wrong (why are we highlighting the same region, twice, with different rules?).

Testing it now from the first commit in this PR (3098b61), i got it misbehaving with these rules:

rules:
    - todo: "xxx"
    - comment:
        start: "//"
        end: "$"
        rules:
            - identifier:
                start: "<"
                end: ">"
                rules:
                    - todo: "todo"

which given the following text:

xxx
// xxx < xxx >

highlights everything as you would expect, except for the last xxx, which gets todo instead of identifier, which it clearly shouldn't. (Because the "comment" region calls highlightEmptyRegion which highlights xxx as "todo".)

I'll split it out into it's own commit later today.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, thanks.

@redsti-github

Copy link
Copy Markdown
Contributor Author

I updated up the commit messages a bit, hopefully they are less confusing now.
Also tests are now in the associated commit instead of a seperate one.

(Thanks dmaluka for the feedback!, i'm still new to this contributing to open source thing)

Comment thread pkg/highlight/highlighter_test.go Resolved
Comment thread pkg/highlight/highlighter_test.go Resolved
Comment thread pkg/highlight/highlighter.go Resolved
@dmaluka

dmaluka commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator

Could you also wrap the lines in the commit messages e.g. at max 72 characters each, to make them plain-text friendly?

@redsti-github
redsti-github force-pushed the fix/highlight branch 2 times, most recently from 051bbda to b7bc9eb Compare November 15, 2026 11:30
@redsti-github

Copy link
Copy Markdown
Contributor Author

I added your changes.

And, sigh, made a slight mess fixing merge conflicts (but i think i fixed it).
One of these days i'll learn how to deal with merge conflicts properly...

@dmaluka

dmaluka commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator

Nit: regarding the commit message of the 1st commit: as you can see for example at https://github.com/micro-editor/micro/pull/4022/commits, the commit title is cut at "...line lengths and", since you wrapped the title line as well, not just the body.

Git commit messages conventionally consist of a single title line (not wrapped) followed by a blank line and a message body (preferably wrapped, usually at 72 characters).

I.e. we don't wrap commit titles but simply make them not too long (preferably, again, max 72 characters). In this particular case, "patterns within regions no longer fail on specific line lengths and columns (#4018)" is not extremely long (83 characters), so I'd be fine with keeping it as is. OTOH we can easily make it shorter by making it less verbose.

In particular, no need for this "(#4018)". Those "(#...)" references that you see throughout micro's git log are not added by commit authors, they are added automatically by Github when merging PRs, and they reference PRs themselves, not issues fixed by those PRs.

Instead, add "Fixes #4018" to the commit message body. BTW that will also make Github automatically close #4018 when this PR is merged.

While we're at it, in commit messages it is also often preferred to describe what the patch does (i.e. what it fixes, or improves, or implements, etc), in the imperative mood, e.g.: "Fix smth ...". So we can improve these 3 commit titles e.g. as follows:

  1. "Fix patterns within regions failing on specific line lengths and columns"
  2. "Fix nested regions causing skipping highlighting parent region"
  3. "Don't needlessly match region patterns outside the region"

redsti added 4 commits November 16, 2026 13:58
Fixes micro-editor#4018
`start` is an index into the entire line, while `line` is from the start
of the region to the end of the line (and so is `endLoc`).
Region body should only be highlighted with that region's rules (call to
`highlightRegion` below removed line), not with the empty region rules.
Previously `<a style="abcd" size=5></a>` would not highlight `style` in
the `identifier` colour, because highlighting the nested region `"abcd"`
would cause an early return before code for highlighting the current
region was reached.
Mostly just a small optimisation; currently, regions will attempt to
highlight everything from the start of the region to the end of the
line, and then discard everything that was past the end of the region.
This commit makes rules scan only inside of the region.
As a side effect, `$` now matches the end of a region as well as the end
of a line.
I'm not aware of any current syntax rules that this would break.
It also adds some nice symmetry between `$` and `^`, as `^` was already
matching to the start of regions.
@redsti-github

Copy link
Copy Markdown
Contributor Author

Those are much better commit messages, thank you!

@JoeKar
JoeKar merged commit 3281b6c into micro-editor:master Nov 17, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants