DEV: Convert the MediaWiki inline parser to byte-offset jump scanning

The inline parser walked its input one character at a time — a String
allocation per character (909/post on the corpus) — and
consecutive_apostrophes_at sliced the entire remaining input on every
probe (374/post, O(n²) bytes across an apostrophe run). Character
indices also made every index/slice O(pos) on multibyte input.

The main loop now jumps between interesting bytes (' [ <) with one
byteindex search and copies each skipped span in a single byteslice;
apostrophe runs are counted with getbyte; the HTML tag match anchors
at the cursor via a \G regex with byteindex instead of slicing the
rest of the input; and all offsets are byte offsets (boundary
invariant: jumps land on ASCII matches, advances step over ASCII
bytes or whole matches).

Corpus bench: mw_parse 94.6 → 32.0 µs/post ASCII and 137.5 → 40.4
multibyte; 1583 → ~170 objects/post. Output is unchanged — I diffed
the rendered Markdown of all 400 corpus posts plus hand-picked Unicode
edge cases (Cyrillic apostrophe names, French elisions, Japanese link
targets, unterminated constructs after multibyte text) between the old
and new implementation: zero differences, including unknown_tags.

New specs place multibyte text before every construct, shaped so a
byte/char-index mixup would land on an alphanumeric byte and change
the outcome. Two leftovers mutant caught: the failure-branch
`@pos = start + count` reset in parse_apostrophe_formatting is gone
(the rewritten collect_until_apostrophes no longer trashes @pos on
failure, so the reset — and the start parameter — were dead), and
consecutive_apostrophes_at's `count = 0` seed goes on the ignore
list (count = 1 is equivalent because every call site sits on an
apostrophe byte).

Author: gschlager

lib/markbridge/parsers/media_wiki/inline_parser.rb

@@ -7,6 +7,14 @@ module MediaWiki
       # Handles bold ('''), italic (''), links ([[...]]), external links ([...]),
       # and HTML inline tags via an InlineTagRegistry.
       #
+      # The parser works in *byte* offsets (byteindex/byteslice/getbyte):
+      # character indices are O(pos) on multibyte input in CRuby, and
+      # per-character probes allocate a String each. The main loop jumps
+      # between interesting bytes with a single regex search and copies
+      # the skipped span in one slice. Invariant: +@pos+ always sits on a
+      # character boundary — jumps land on ASCII matches and advances
+      # step over ASCII bytes or whole matches.
+      #
       # @example With custom registry
       #   registry = InlineTagRegistry.build_from_default do |r|
       #     r.register("mark", :formatting, AST::Bold)
@@ -15,6 +23,19 @@ module MediaWiki
       class InlineParser
         MAX_INLINE_DEPTH = 20
 
+        # Bytes the main loop reacts to: apostrophes ('), link openers ([)
+        # and HTML tag openers (<). ASCII-only, so a byteindex match always
+        # lands on a character boundary.
+        INTERESTING = /['\[<]/
+        APOSTROPHE = 39 # '
+        BRACKET_OPEN = 91 # [
+        private_constant :INTERESTING, :APOSTROPHE, :BRACKET_OPEN
+
+        # Matches an HTML-like tag at the search position (\G with
+        # byteindex anchors at the offset).
+        HTML_TAG_AT_CURSOR = %r{\G<(/?)([a-z]+)(?: [^>]*)?\s*(/?)>}i
+        private_constant :HTML_TAG_AT_CURSOR
+
         # @return [Hash{String => Integer}] tag-name → occurrence count for
         #   HTML-like inline tags whose names are not registered. Shared
         #   with nested InlineParser instances so depth-recursive parses
@@ -34,56 +55,66 @@ def initialize(handlers: nil, depth: 0, unknown_tags: nil)
         def parse(text, parent:)
           @input = text
           @pos = 0
-          @length = text.length
+          @length = text.bytesize
           @parent = parent
           @text_buffer = +""
 
-          while @pos < @length
-            char = @input[@pos]
-
-            case char
-            when "'"
-              consecutive_apostrophes_at(@pos) >= 2 ? parse_bold_italic : append_literal(char)
-            when "["
-              flush_text
-              @input[@pos + 1] == "[" ? parse_internal_link : parse_external_link
-            when "<"
-              flush_text
-              parse_html_tag
-            else
-              append_literal(char)
-            end
+          while (span_end = @input.byteindex(INTERESTING, @pos))
+            # Unconditional on purpose: when the interesting byte sits at
+            # @pos the slice is empty and both lines are no-ops.
+            @text_buffer << @input.byteslice(@pos, span_end - @pos)
+            @pos = span_end
+
+            dispatch_interesting_byte
           end
 
+          # Trailing text after the last interesting byte; appending the
+          # empty slice at end-of-input is a no-op.
+          @text_buffer << @input.byteslice(@pos, @length - @pos)
           flush_text
         end
 
         private
 
-        def append_literal(char)
-          @text_buffer << char
-          @pos += 1
+        # Precondition: the byte at +@pos+ matched INTERESTING.
+        def dispatch_interesting_byte
+          case @input.getbyte(@pos)
+          when APOSTROPHE
+            if consecutive_apostrophes_at(@pos) >= 2
+              parse_bold_italic
+            else
+              @text_buffer << "'"
+              @pos += 1
+            end
+          when BRACKET_OPEN
+            flush_text
+            @input.getbyte(@pos + 1) == BRACKET_OPEN ? parse_internal_link : parse_external_link
+          else # "<" — the only remaining INTERESTING byte
+            flush_text
+            parse_html_tag
+          end
         end
 
-        # Precondition: caller has verified @input[@pos..@pos+1] is "''".
+        # Precondition: caller has verified at least two apostrophes at @pos.
         def parse_bold_italic
-          start = @pos
           count = [consecutive_apostrophes_at(@pos), 5].min
           flush_text
           @pos += count
-          parse_apostrophe_formatting(count, start)
+          parse_apostrophe_formatting(count)
         end
 
         # Parse apostrophe-delimited formatting (bold, italic, or bold+italic).
+        # Entered with @pos just past the opening apostrophes.
         #
         # @param apostrophe_count [Integer] number of apostrophes (2, 3, or 5)
-        # @param start [Integer] position before the opening apostrophes
-        def parse_apostrophe_formatting(apostrophe_count, start)
+        def parse_apostrophe_formatting(apostrophe_count)
           content = collect_until_apostrophes(apostrophe_count)
 
           unless content
+            # @pos already sits just past the opening apostrophes
+            # (start + apostrophe_count): the caller advanced it, and
+            # collect_until_apostrophes leaves it untouched on failure.
             @text_buffer << ("'" * apostrophe_count)
-            @pos = start + apostrophe_count
             return
           end
 
@@ -124,44 +155,50 @@ def parse_inner_content(content, parent:)
           ).parse(content, parent:)
         end
 
-        # Collect text until we find n consecutive apostrophes.
-        # Returns the collected content string or nil if not found.
+        # Collect text until we find n consecutive apostrophes, hopping
+        # from apostrophe run to apostrophe run instead of scanning per
+        # position. Returns the collected content string, or nil (the
+        # exhausted while loop) if no closing run exists.
         #
         # @param count [Integer] number of consecutive apostrophes to match
         # @return [String, nil]
         def collect_until_apostrophes(count)
           start = @pos
-          while @pos < @length
-            if consecutive_apostrophes_at(@pos) >= count
-              content = @input[start...@pos]
-              @pos += count
+          probe = @pos
+          while (index = @input.byteindex("'", probe))
+            run = consecutive_apostrophes_at(index)
+            if run >= count
+              content = @input.byteslice(start, index - start)
+              @pos = index + count
               return content
             end
-            @pos += 1
+            probe = index + run
           end
         end
 
         # Count consecutive apostrophes starting at position.
+        # getbyte past end-of-input returns nil, which ends the run.
         #
         # @param pos [Integer]
         # @return [Integer]
         def consecutive_apostrophes_at(pos)
-          @input[pos..].each_char.take_while { |c| c == "'" }.length
+          count = 0
+          count += 1 while @input.getbyte(pos + count) == APOSTROPHE
+          count
         end
 
         # Parse [[internal link]] or [[target|display text]].
         def parse_internal_link
           @pos += 2 # skip [[
           start = @pos
 
-          # Find closing ]]
-          close_pos = @input.index("]]", @pos)
+          close_pos = @input.byteindex("]]", @pos)
           unless close_pos
             @text_buffer << "[["
             return
           end
 
-          content = @input[start...close_pos]
+          content = @input.byteslice(start, close_pos - start)
           @pos = close_pos + 2
 
           target, display = content.split("|", 2)
@@ -178,14 +215,13 @@ def parse_external_link
           @pos += 1 # skip [
           start = @pos
 
-          # Find closing ]
-          close_pos = @input.index("]", @pos)
+          close_pos = @input.byteindex("]", @pos)
           unless close_pos
             @text_buffer << "["
             return
           end
 
-          content = @input[start...close_pos]
+          content = @input.byteslice(start, close_pos - start)
           @pos = close_pos + 1
 
           # Split on first space: URL followed by optional display text
@@ -199,13 +235,13 @@ def parse_external_link
         end
 
         def parse_html_tag
-          tag_match = @input[@pos..].match(%r{\A<(/?)([a-z]+)(?: [^>]*)?\s*(/?)>}i)
-          unless tag_match
+          unless @input.byteindex(HTML_TAG_AT_CURSOR, @pos)
             @text_buffer << "<"
             @pos += 1
             return
           end
 
+          tag_match = Regexp.last_match
           full_match = tag_match[0]
           closing = !tag_match[1].empty?
           self_closing = !tag_match[3].empty?
@@ -238,25 +274,25 @@ def dispatch_html_tag(entry, tag_name, full_match)
           when :formatting
             handle_paired_tag(tag_name, full_match, entry.element_class)
           when :self_closing
-            @pos += full_match.length
+            @pos += full_match.bytesize
             @parent << entry.element_class.new
           end
         end
 
         # Advance position and buffer the match as literal text.
         def advance_as_text(full_match)
           @text_buffer << full_match
-          @pos += full_match.length
+          @pos += full_match.bytesize
         end
 
         # Handle <nowiki>...</nowiki> - preserves content as literal text.
         def handle_nowiki_tag(full_match)
-          @pos += full_match.length
-          close_pos = @input.index("</nowiki>", @pos)
+          @pos += full_match.bytesize
+          close_pos = @input.byteindex("</nowiki>", @pos)
 
           if close_pos
-            @text_buffer << @input[@pos...close_pos]
-            @pos = close_pos + "</nowiki>".length
+            @text_buffer << @input.byteslice(@pos, close_pos - @pos)
+            @pos = close_pos + "</nowiki>".bytesize
           else
             @text_buffer << full_match
           end
@@ -265,15 +301,15 @@ def handle_nowiki_tag(full_match)
         # Handle paired raw tags like <code>...</code> and <pre>...</pre>.
         # Content inside is not parsed for wiki markup.
         def handle_paired_raw_tag(tag_name, full_match, element_class)
-          @pos += full_match.length
+          @pos += full_match.bytesize
           close_tag = "</#{tag_name}>"
-          close_pos = @input.index(close_tag, @pos)
+          close_pos = @input.byteindex(close_tag, @pos)
 
           if close_pos
             element = element_class.new
-            element << AST::Text.new(@input[@pos...close_pos])
+            element << AST::Text.new(@input.byteslice(@pos, close_pos - @pos))
             @parent << element
-            @pos = close_pos + close_tag.length
+            @pos = close_pos + close_tag.bytesize
           else
             @text_buffer << full_match
           end
@@ -282,26 +318,28 @@ def handle_paired_raw_tag(tag_name, full_match, element_class)
         # Handle paired formatting tags like <s>, <u>, <sup>, <sub>.
         # Content inside IS parsed for wiki markup.
         def handle_paired_tag(tag_name, full_match, element_class)
-          @pos += full_match.length
+          @pos += full_match.bytesize
           close_tag = "</#{tag_name}>"
-          close_pos = @input.index(close_tag, @pos)
+          close_pos = @input.byteindex(close_tag, @pos)
 
           if close_pos
             element = element_class.new
-            parse_inner_content(@input[@pos...close_pos], parent: element)
+            parse_inner_content(@input.byteslice(@pos, close_pos - @pos), parent: element)
             @parent << element
-            @pos = close_pos + close_tag.length
+            @pos = close_pos + close_tag.bytesize
           else
             @text_buffer << full_match
           end
         end
 
         # Flush accumulated text buffer to the parent as a Text node.
+        # AST::Text copies the (mutable) buffer, so clearing it afterwards
+        # reuses the allocated capacity for the next span.
         def flush_text
           return if @text_buffer.empty?
 
           @parent << AST::Text.new(@text_buffer)
-          @text_buffer = +""
+          @text_buffer.clear
         end
       end
     end

mutant.yml

@@ -216,6 +216,15 @@ matcher:
     - Markbridge::Parsers::MediaWiki::InlineParser#initialize
     - Markbridge::Parsers::MediaWiki::InlineParser#parse_inner_content
 
+    # InlineParser#consecutive_apostrophes_at's `count = 0` seed. The
+    # `count = 1` mutation is equivalent because every call site sits
+    # on an apostrophe byte (the dispatch case arm, byteindex("'")
+    # results, and parse_bold_italic's precondition), so skipping the
+    # zeroth-byte check changes nothing. Killing it would require
+    # calling the helper on a non-apostrophe position, which no public
+    # input can produce.
+    - Markbridge::Parsers::MediaWiki::InlineParser#consecutive_apostrophes_at
+
     # MediaWiki Parser#initialize's block_given? conditional and the
     # InlineTagRegistry kwarg plumbing. Mutations on `if block_given?`
     # (`if true` / `if false`) and on the `inline_tag_registry ||`