I extracted this "JS porter" code to actually verify this, and ran it against Snowball's test suite, which reveals more problems. For example, its undoubling rule is buggy and it stems wrapped to wrapp, while both "porter" and "english" stem it to "wrap", so it's a buggy implementation compared to either. That means a query for wrapped will fail to match wrapped in a document.

@ojwb I'm having difficulty replicating this part of the report; I find that the EN-language JS stemmer and Python equivalent both stem the word wrapped to wrap, meaning that wrapped remains valid as a findable query token.

I'm having difficulty replicating this part of the report

I looked into this, and it was caused by me failing to unescape the jsporter code fully. I had fixed up two long regexp lines which had been wrapped and \ inserted, but I'd failed to spot where \\ was escaped to \\\\ in another regexp. Fixing that I do indeed get wrap for wrapped - sorry for the misleading information there.

The outputs still don't match for other cases though. Repeating the test there are 68 differences compared to Snowball's "porter" algorithm which is not nearly as bad, but still not ideal. (Compared to the "english" algorithm, there are actually 2176 differences so jsporter is definitely closer to "porter".)

The main differences are (porter vs jsporter):

• -ibly (e.g. audibly -> audibli vs audibl)
• -ology (e.g. tautology -> tautologi vs tautolog)
• -s on very short inputs (e.g. ms -> m vs ms)

I'd actually say jsporter considered in by itself does better in all these cases because it conflates e.g. audibly/audible and tautology/tautological, and doesn't conflate ms/m, but it's problematic to use a different stemmer at index and search time because searches for the affected words won't match themselves in the text that was indexed.

Snowball's "english" stemmer has these improvements too, but also various others, so I'd still suggest standardising on that for both Python and Javascript. It's likely to be easier than adjusting jsporter to exactly match Snowball's "porter" stemmer, while giving the benefit of better English stemming.

Here's the patched stemwords.js with the correctly unescaped jsporter code in case it is useful to you for testing (I had to add .txt to the filename for github to allow me to attach it):

jsporter-stemwords.js.txt

You can test it on a single word like so:

$ echo wrapped|node jsporter-stemwords.js.txt -i /dev/stdin -o /dev/stdout
wrap

I looked into this, and it was caused by me failing to unescape the jsporter code fully. I had fixed up two long regexp lines which had been wrapped and \ inserted, but I'd failed to spot where \\ was escaped to \\\\ in another regexp. Fixing that I do indeed get wrap for wrapped - sorry for the misleading information there.

That's OK - thank you for the report, and for checking that! I'm on-board with migrating from the porter to more-developed English stemmer and also updating all of the bundled stemmers (both Python and JS) - especially after reading through some of the recent changelog entries.

I'll try to get around to starting the update within the next week or so.

The recent issue with our 3.0.0 release caused me to notice that you're using our "porter" stemmer, which is really still provided only for academic interest.

Thanks for letting us know. I've tried to work out why I didn't use English when I changed to use snowballstemmer in 2022, and encountered a few challenges with the snowball documentation, which I've detailed at the end.

Looking at master/sphinx/search/en.py I see there's also a "JS porter" implementation which looks like a hand-written implementation, and my initial thought that "oh, they're using the older and less good porter stemmer because they need to be compatible with that", but looking at the "JS porter" javascript code, it isn't actually an implementation of Porter's 1980 algorithm. For example, line 48 has logi: 'log' which is one of the additional rules added in "porter2". My best guess is it's a hand-written implementation of an early version of "porter2".

I can't work out the provenance of this parser, it's rather strange. It appears to date back to the first revision of proto-Sphinx (then called py-rest-doc). @birkenfeld may know, but it was nearly 20 years ago!

I'd suggest the best way to resolve this would be to switch from "porter" to "english" (because the latter has improvements from 45 years of experience using the original so is a significantly better stemmer) and replace this "JS porter" implementation with a Javascript version of the same stemmer generated by Snowball. Snowball's upstream testsuite should ensure these produce the same stems (at least if you take them from the same upstream Snowball release, but evolution is slow at this point so even version skew is not going to give you different stems for 2.7% of words).

This makes sense, I'll propose a PR.

A

Snowball documentation troubles:

I tried to find the documentation, but couldn't easily find the link on snowball's PyPI page, I eventually found https://snowballstem.org/algorithms/. On the list of English stemmers, the first three are Porter/Lovins, with the main English stemmer 'buried' in the list. The brackets are unclear, it turns out they mean 'not recommended', but perhaps it might be an improvement to say 'academic interest only: blah', or list English first, or put it in boldface?

The page for the english stemmer (https://snowballstem.org/algorithms/english/stemmer.html) also contains no usage exemplars, so somewhat unclear if I should use snowballstemmer.stemmer('porter2') or 'english' etc.

Haha, sorry, I have no idea anymore :) It may even have been @mitsuhiko, he did a lot of the early HTML/JS part.