8350459: MontgomeryIntegerPolynomialP256 multiply intrinsic with AVX2 on x86_64

[vpaprotsk]

Add AVX2 montgomery multiplication intrinsic. (About 60-80% gain)

Also add reduction to existing AVX512 multiplication (this was left-over from #19893 where a quick fix was required). This is mostly for cleanup, but there is about 1-2% gain.

Before (no AVX512)

Benchmark                        (algorithm)  (dataSize)  (keyLength)  (provider)   Mode  Cnt      Score     Error  Units
SignatureBench.ECDSA.sign    SHA256withECDSA        1024          256              thrpt   40   3720.589 ±  17.879  ops/s
SignatureBench.ECDSA.sign    SHA256withECDSA       16384          256              thrpt   40   3605.940 ±  15.807  ops/s
SignatureBench.ECDSA.verify  SHA256withECDSA        1024          256              thrpt   40   1076.502 ±   4.190  ops/s
SignatureBench.ECDSA.verify  SHA256withECDSA       16384          256              thrpt   40   1069.624 ±   2.484  ops/s
Benchmark                             (algorithm)  (keyLength)  (kpgAlgorithm)  (provider)   Mode  Cnt     Score   Error  Units
KeyAgreementBench.EC.generateSecret          ECDH          256              EC              thrpt   40   830.448 ± 2.285  ops/s

After (with AVX2)

Benchmark                        (algorithm)  (dataSize)  (keyLength)  (provider)   Mode  Cnt      Score     Error  Units
SignatureBench.ECDSA.sign    SHA256withECDSA        1024          256              thrpt   40   6000.496 ±  39.923  ops/s
SignatureBench.ECDSA.sign    SHA256withECDSA       16384          256              thrpt   40   5739.878 ±  34.838  ops/s
SignatureBench.ECDSA.verify  SHA256withECDSA        1024          256              thrpt   40   1942.437 ±  12.179  ops/s
SignatureBench.ECDSA.verify  SHA256withECDSA       16384          256              thrpt   40   1921.770 ±   8.992  ops/s
Benchmark                             (algorithm)  (keyLength)  (kpgAlgorithm)  (provider)   Mode  Cnt     Score   Error  Units
KeyAgreementBench.EC.generateSecret          ECDH          256              EC              thrpt   40  1399.761 ± 6.238  ops/s

Before (with AVX512):

Benchmark                        (algorithm)  (dataSize)  (keyLength)  (provider)   Mode  Cnt       Score     Error  Units
SignatureBench.ECDSA.sign    SHA256withECDSA        1024          256              thrpt   40    9621.950 ±  27.260  ops/s
SignatureBench.ECDSA.sign    SHA256withECDSA       16384          256              thrpt   40    8975.654 ±  26.707  ops/s
SignatureBench.ECDSA.verify  SHA256withECDSA        1024          256              thrpt   40    3112.945 ±  12.930  ops/s
SignatureBench.ECDSA.verify  SHA256withECDSA       16384          256              thrpt   40    3039.183 ±  12.362  ops/s
Benchmark                             (algorithm)  (keyLength)  (kpgAlgorithm)  (provider)   Mode  Cnt     Score    Error  Units
KeyAgreementBench.EC.generateSecret          ECDH          256              EC              thrpt   40  2248.987 ±  7.427  ops/s

After (with AVX512):

Benchmark                        (algorithm)  (dataSize)  (keyLength)  (provider)   Mode  Cnt       Score     Error  Units
SignatureBench.ECDSA.sign    SHA256withECDSA        1024          256              thrpt   40    9815.713 ±  23.455  ops/s
SignatureBench.ECDSA.sign    SHA256withECDSA       16384          256              thrpt   40    9136.786 ±  27.747  ops/s
SignatureBench.ECDSA.verify  SHA256withECDSA        1024          256              thrpt   40    3167.702 ±  13.331  ops/s
SignatureBench.ECDSA.verify  SHA256withECDSA       16384          256              thrpt   40    3090.053 ±  12.925  ops/s
Benchmark                             (algorithm)  (keyLength)  (kpgAlgorithm)  (provider)   Mode  Cnt     Score    Error  Units
KeyAgreementBench.EC.generateSecret          ECDH          256              EC              thrpt   40  2278.031 ±  6.971  ops/s

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8350459: MontgomeryIntegerPolynomialP256 multiply intrinsic with AVX2 on x86_64 (Enhancement - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/23719/head:pull/23719
$ git checkout pull/23719

Update a local copy of the PR:
$ git checkout pull/23719
$ git pull https://git.openjdk.org/jdk.git pull/23719/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 23719

View PR using the GUI difftool:
$ git pr show -t 23719

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/23719.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back vpaprotski! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@vpaprotsk The following labels will be automatically applied to this pull request:

• hotspot
• security

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 00: Full (e0803952)