Prover/dev small fields #1301
Draft
Prover/dev small fields #1301
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* feat(precompiles): parametrize better the nb of input per instance * test(affinity): adds a test with the affinity * fix(limitless): debug the affinity system * feat(recursion): uses the fixed number of row parameter * fix(gnark-verifier): fix the gnark verifier * fix(lane): change a constraint in the lane package to make it work * chores(zkevm.bin): updates the zkevm.bin to work with d855c03 * fix(permutation); fix compilation error coming from conflict and clean the folder * feat(mimc): implements an external hasher builder * clean(plonk-in-wizard): reorganize the package handling plonk in wizard * reorg(counterPrecomputed): creates a specific type of precomputed columns that caches itself * feat(plonk): implements the external-checker feature * feat(mimc): adds a selector feature for the MiMC query * feat(distribution): uses the external hasher to build the recursion circuit. * feat(distribution): grouping the LPP modules * feat(test): adds back the LPP part of the tests * feat(wizard): adds verify until round * fix(arcane): remove the NormalizeOffset * clean(dist): sanity check the intermediate proof * chores(gnark): updates gnark version to latest main and port the values * feat(stackframe): implements a helper to return callframes of the calling context * perf(smartvectors): adds a routine to optimize the right padded vectors * org(factory): hasher factory does not require an API to return a hasher * feat(modexp): adds a limit for the modexp 4096 * fix the hash-checker implements the predivision optimize the logderivative implementation * feat(smart-vectors): implements iterators for smart-vectors * feat(field): adds a FromBool method * perf(polyeval): adds a fast-path to PolyEval when all inputs are constants where we don't need to allocate * feat(smartvectors): adds iterators and GetPtr functionalities to the interface * perf(mimc): revamp the mimc compiler to be more space-efficient * feat(dedicated): adds a bit decomposition wizard * feat(ternary): adds a ternary wizard * feat(hashing): adds an horizontal hash dedicated wizard * feat(merkle): adds a flat-merkle proof dedicated wizard * feat(utils): adds SpliceExact Bytes32.SetField and a few iterators functions * feat(dedicated): adds a [Last] method * perf(zkevm): makes the vector allocation faster * perf(symbolic): skips the copy-slice when the result of a chunk is full-zero * perf(logerivsum): makes the log derivative sum assignment more efficient * feat(dist): improves the distributed wizard * feat: review the public input fetching * feat(ts-fetcher): duplicate columns to have clean module separation * fix(maintest): runs the normal proof more efficiently * chores(gkr): reflect the API changes in the gkr frontend to gkrmimc * chores(go.mod): update gnark version to v0.12.1-0.20250409125015-1e252ace2ab9 * chores(tests): skip the big tests of the distributed prover so that they are not run by the CI * fix(lint): fix ineffectual assignment * back with the old linter * fix(tests): fix the padding sanity-checks in the limb decomposition
* replace proverstep with main prover step * lookup tests pass * stitchsplit tests fail * fix splitter tests * stitchsplit compiler tests pass * univariate compiler all tests pass * vortex compiler tests pass * bigrange tests pass * dedicated/expr_handle refactor * protocol/dedicated/functionals tests pass * protocol/dedicated/merkle pass * all tests in protocol/dedicated pass * remove unwanted comments * self recursion last tests fail * all protocol tests pass * all tests pass * Transform splitter ctx prover to proverAction interface
* init serialization doc * commit progress * commit progress * all tests pass * refactor serialization.go * refactor compiled iop * parallelize se-deser compiled-iop * all tests pass * update comments * split cbor encoder and decoder * update go.mod
…all-field-symbolic-refactored-branch-2
* cleanup batch lagrante, add panic for different size poly * refactor: call batch lagrange ext from gnark-crypto * Pre-allocate data in batch lagrange * add sm_mixed.BatchEvaluateLagrange to handle mixed polys * add sm_mixed.BatchEvaluateLagrange to handle mixed polys * call mixed version batch * cleanup BatchEvaluateLagrange * call gnark batch functions * Revert "call gnark batch functions" This reverts commit a9b9c95. * revert one commit, update gnark-crypto * update gnark-crypto * update version * cleanup lagrange * rename lagrange basing on the poly value * keep BatchEvaluateLagrangeMixed for split * cleanup some poly Eval * typo * remove redudant lines * add panic err * fix error handling and test call * update gnark-crypto version, add avx implementation for innerproduct by element * update gnark-crypto version * update go.sum --------- Signed-off-by: Yao J. Galteland <[email protected]>
* feat: take constant into accounts when splitting * feat: cleanup reedsolomon, fft and started quotient * perf: better memory management in quotient * perf: remove greedy pools * perf: hunting innefficiencies * perf: less go routines in quotient * refactor: kill the mempool package * fix: missing poly in hint * refactor: remove more pool related stuff
…field (#1457) * feat: add experimental vector horner eval * perf: more minor perf adjustements * refactor: use vector e4 from gnark-crypto * chore: update gnark-crypto
* perf: various improvements * style: clean up * perf: adjustements
* feat: add poseidon2, test consistency * refactor tree * feat: add a poseidon2 consistency test, Write one element at a time * feat: add panic msg in smartvectors * fix(vortex): fix two tests in vortex compiler * fix(ringsis): fix param test due to param change * fix (sv): fix TestTryReduceSizeRight * docs: comment the factories test to fix after the poseidon2 package is done, comment the gnark test * refactor: Bytes32ToHash * minor * fix(hash input): we never run without a hash, add panic msg * renaming(MIMC) * renaming: rename mimc to Poseidon2 in vortex package * fix mimc compiler test, add InsertPoseidon2 * refactor: poseidon query, add testcases * fix: constant zero tests * all test in ListOfPoseidon2Testcase pass * add Poseidon2Sponge * feat: add poseidon2 RoundKey, defineMatMulExternalInPlace, defineRoundInternal, defineAddRoundKeyAndSBox * update gnark-crypto version, implement poseidon2 constraints, poseidon2 helper functions * feat: assign intermediate value columns * fix basic errors in poseidon2 compiler * renaming rounds * refactor: adjust locations of columns and rounds, adjust indexes * add test function to check the consistency of poseidon2 inner functions * fix MatMulInternal_COMPUTATION global constraints * minor edit * fixed the global constraints for addRoundKeys and sBox * fixed feed-forward constraints * cleanup code step 1 * clean up code: remove data conversion * cleanup Reducing memory allocations * cleanup crypto/poseidon2 * refactor smartvectors * clean up * fix a few bugs * refactor: set default leafhash to poseidon2 sponge to aviod data type conversion * refactor: write the whole bytes * uncomment realistic-segment * docs: add descriptions * feat: use Gaudam's perf/e4vec branch * refactor: coin to Ext * fix: dedicated/selector ok * fix: isbase was calling itself endlessly * refactor: coin fext * reedsolomon ok * byte32cmp ok * update gnark-crypto * add tests in makefile * minor * TestEvalCoeff ok * minor * update merkle * cleanup errors in dedicated merkle * merkle test ok * update mimc-->poseidon2 * update mimc-->poseidon2 * flatproof test ok * linea hash update, only the last value check fail * locate the bug * linea hash ok * remove note * linea hash support sponge * cleanup some todo * remove all errors * fix merge errors * fix test errors * further cleanup, add plonk in makefile for test * minor * update gnark version * refactor: pre-allocate segments to Tohash columns in linea hash * reorganize the hash preimages * cleanup linea hash * cleanup the PrepareToHashWitness * debug linearhashandmerkle * cleanup the PrepareToHashWitness, all roots hold 8 columns * implement IntoRegVecSaveAlloc and IntoRegVecSaveAllocBase for base * fix broken test * merge * need the latest gnark * foldouter coin to ext * fold to ext ok * remove all compilation errors in selfrecursion * fix: sis check in selfrecursion ok * fix: selfrecursion rs ok * fix: TestSelfRecursionRandom ok * fix: TestSelfRecursionCommitPrecomputed ok * replace mimc with poseidon2 compiler * replace mimc with poseidon2 compiler * fix: coin --> ext coin * merge * fix: only non-sis ok * cleanup * ualpha consistency ok * clean up some notes * fix: mandatory name in RandLinCombCol * fix: empty round test * fix: npolys increase * add selfrecrusion tests in makefile * cleanup recrusion * feat(eval-coeff): adds support for length one columns * fold getval --> getvalext * cleanup selfrecursion * minor * fix: innerproduct * cleanup test and notes * init param * fix: data type consistency for test * bench: adds a standard benchmark with self-recursion * fix: bugs and resolve comments * fix: eval coeff * fix: ExtraData holds the whole root --------- Co-authored-by: Yao Galteland <[email protected]> Co-authored-by: AlexandreBelling <[email protected]>
…atShamir (#1578) * feat: add poseidon2, test consistency * refactor tree * feat: add a poseidon2 consistency test, Write one element at a time * feat: add panic msg in smartvectors * fix(vortex): fix two tests in vortex compiler * fix(ringsis): fix param test due to param change * fix (sv): fix TestTryReduceSizeRight * docs: comment the factories test to fix after the poseidon2 package is done, comment the gnark test * refactor: Bytes32ToHash * minor * fix(hash input): we never run without a hash, add panic msg * renaming(MIMC) * renaming: rename mimc to Poseidon2 in vortex package * fix mimc compiler test, add InsertPoseidon2 * refactor: poseidon query, add testcases * fix: constant zero tests * all test in ListOfPoseidon2Testcase pass * add Poseidon2Sponge * feat: add poseidon2 RoundKey, defineMatMulExternalInPlace, defineRoundInternal, defineAddRoundKeyAndSBox * update gnark-crypto version, implement poseidon2 constraints, poseidon2 helper functions * feat: assign intermediate value columns * fix basic errors in poseidon2 compiler * renaming rounds * refactor: adjust locations of columns and rounds, adjust indexes * add test function to check the consistency of poseidon2 inner functions * fix MatMulInternal_COMPUTATION global constraints * minor edit * fixed the global constraints for addRoundKeys and sBox * fixed feed-forward constraints * cleanup code step 1 * clean up code: remove data conversion * cleanup Reducing memory allocations * cleanup crypto/poseidon2 * refactor smartvectors * clean up * fix a few bugs * refactor: set default leafhash to poseidon2 sponge to aviod data type conversion * refactor: write the whole bytes * uncomment realistic-segment * docs: add descriptions * feat: use Gaudam's perf/e4vec branch * refactor: coin to Ext * fix: dedicated/selector ok * fix: isbase was calling itself endlessly * refactor: coin fext * reedsolomon ok * byte32cmp ok * update gnark-crypto * add tests in makefile * minor * TestEvalCoeff ok * minor * update merkle * cleanup errors in dedicated merkle * merkle test ok * update mimc-->poseidon2 * update mimc-->poseidon2 * flatproof test ok * linea hash update, only the last value check fail * cleanup hashes * locate the bug * tree * set default hash * minor * linea hash ok * remove note * linea hash support sponge * cleanup some todo * remove all errors * fix merge errors * fix test errors * further cleanup, add plonk in makefile for test * minor * update gnark version * refactor: pre-allocate segments to Tohash columns in linea hash * reorganize the hash preimages * cleanup linea hash * cleanup the PrepareToHashWitness * debug linearhashandmerkle * cleanup the PrepareToHashWitness, all roots hold 8 columns * implement IntoRegVecSaveAlloc and IntoRegVecSaveAllocBase for base * fix broken test * merge * need the latest gnark * foldouter coin to ext * fold to ext ok * remove all compilation errors in selfrecursion * fix: sis check in selfrecursion ok * fix: selfrecursion rs ok * fix: TestSelfRecursionRandom ok * fix: TestSelfRecursionCommitPrecomputed ok * replace mimc with poseidon2 compiler * replace mimc with poseidon2 compiler * fix: coin --> ext coin * merge * fix: only non-sis ok * cleanup * ualpha consistency ok * clean up some notes * fix: mandatory name in RandLinCombCol * fix: empty round test * fix: npolys increase * add selfrecrusion tests in makefile * cleanup recrusion * feat(eval-coeff): adds support for length one columns * fold getval --> getvalext * cleanup selfrecursion * minor * fix: innerproduct * cleanup test and notes * init param * fix: data type consistency for test * feat: add FieldHasher * refactor: fieldhasher * cleanup before fix accumulator * refactor: dedicated merkle to fit with the fieldhasher * fix accumulator, wrap hash.Hash in poseidon2hasher * fix: accumulator/operation_test * fix: fix broken merkle tests * cleanup Bytes32 * cleanup Bytes32 * cleanup naming of poseiond2 and hasher * minor * fix merge * feat: adapt fielderhasher * fix: fieldhasher in vortex * docs : cleanup descriptions in fieldhasher * rename Hasher * fix: broken tests * refactor: replace NewMerkleDamgardHasher to fieldhasher * refactor: integrate fieldhasher into FS * cleanup functions, fix UpdateGeneric * refactor: directly sumelements remove some writeelements to reduce the memory * refactor: vector arena, convert bytes to field.Elements * Revert "refactor: vector arena, convert bytes to field.Elements" This reverts commit 239da5d. * revert commit * fix: ai bugs * minor * fix bugs in write and simplify sum * direct sum, without write * fix: Sum compute the hash of msg, add description * remove fieldhasher interface for poseidon2, directly using struct * refactor: remove coin.FieldFromSeed * remove coin.Field * remove GetRandomCoinField * remove FS.RandomField --------- Co-authored-by: Yao Galteland <[email protected]> Co-authored-by: AlexandreBelling <[email protected]>
* bench: adds a standard benchmark with self-recursion * perf(poseidon2): optimize the poseidon2 compiler to create less columns and assign faster * fix: columns are not expressions by default * perf: optimize RS when rho is not 2 * clean the selfrecursion benchmark * change the small field parameters --------- Signed-off-by: AlexandreBelling <[email protected]> Co-authored-by: Copilot <[email protected]>
* feat: add poseidon2, test consistency * refactor tree * feat: add a poseidon2 consistency test, Write one element at a time * feat: add panic msg in smartvectors * fix(vortex): fix two tests in vortex compiler * fix(ringsis): fix param test due to param change * fix (sv): fix TestTryReduceSizeRight * docs: comment the factories test to fix after the poseidon2 package is done, comment the gnark test * refactor: Bytes32ToHash * minor * fix(hash input): we never run without a hash, add panic msg * renaming(MIMC) * renaming: rename mimc to Poseidon2 in vortex package * fix mimc compiler test, add InsertPoseidon2 * refactor: poseidon query, add testcases * fix: constant zero tests * all test in ListOfPoseidon2Testcase pass * add Poseidon2Sponge * feat: add poseidon2 RoundKey, defineMatMulExternalInPlace, defineRoundInternal, defineAddRoundKeyAndSBox * update gnark-crypto version, implement poseidon2 constraints, poseidon2 helper functions * feat: assign intermediate value columns * fix basic errors in poseidon2 compiler * renaming rounds * refactor: adjust locations of columns and rounds, adjust indexes * add test function to check the consistency of poseidon2 inner functions * fix MatMulInternal_COMPUTATION global constraints * minor edit * fixed the global constraints for addRoundKeys and sBox * fixed feed-forward constraints * cleanup code step 1 * clean up code: remove data conversion * cleanup Reducing memory allocations * cleanup crypto/poseidon2 * refactor smartvectors * clean up * fix a few bugs * refactor: set default leafhash to poseidon2 sponge to aviod data type conversion * refactor: write the whole bytes * uncomment realistic-segment * docs: add descriptions * feat: use Gaudam's perf/e4vec branch * refactor: coin to Ext * fix: dedicated/selector ok * fix: isbase was calling itself endlessly * refactor: coin fext * reedsolomon ok * byte32cmp ok * update gnark-crypto * add tests in makefile * minor * TestEvalCoeff ok * minor * update merkle * cleanup errors in dedicated merkle * merkle test ok * update mimc-->poseidon2 * update mimc-->poseidon2 * flatproof test ok * linea hash update, only the last value check fail * cleanup hashes * locate the bug * tree * set default hash * minor * linea hash ok * remove note * linea hash support sponge * cleanup some todo * remove all errors * fix merge errors * fix test errors * further cleanup, add plonk in makefile for test * minor * update gnark version * refactor: pre-allocate segments to Tohash columns in linea hash * reorganize the hash preimages * cleanup linea hash * cleanup the PrepareToHashWitness * debug linearhashandmerkle * cleanup the PrepareToHashWitness, all roots hold 8 columns * implement IntoRegVecSaveAlloc and IntoRegVecSaveAllocBase for base * fix broken test * merge * need the latest gnark * foldouter coin to ext * fold to ext ok * remove all compilation errors in selfrecursion * fix: sis check in selfrecursion ok * fix: selfrecursion rs ok * fix: TestSelfRecursionRandom ok * fix: TestSelfRecursionCommitPrecomputed ok * replace mimc with poseidon2 compiler * replace mimc with poseidon2 compiler * fix: coin --> ext coin * merge * fix: only non-sis ok * cleanup * ualpha consistency ok * clean up some notes * fix: mandatory name in RandLinCombCol * fix: empty round test * fix: npolys increase * add selfrecrusion tests in makefile * cleanup recrusion * feat(eval-coeff): adds support for length one columns * fold getval --> getvalext * cleanup selfrecursion * minor * fix: innerproduct * cleanup test and notes * init param * fix: data type consistency for test * feat: add FieldHasher * refactor: fieldhasher * cleanup before fix accumulator * refactor: dedicated merkle to fit with the fieldhasher * fix accumulator, wrap hash.Hash in poseidon2hasher * fix: accumulator/operation_test * fix: fix broken merkle tests * init fieldhasher2 * refactor: fieldhasher * before fix fiatshamir * adapt poseidon2hasher * merge and fix tests * docs: cleanup description * adapt the hash from Thomas * convert to pointer * add reset * cleanup * refactor FS * fix and adapt RandomManyIntegers * overwrite MD Write and Sum * cleanup: move poseidon2fieldhasher to crypto/poseidon2 * delete mimc * cleanup RandomManyIntegers --------- Co-authored-by: Yao Galteland <[email protected]> Co-authored-by: AlexandreBelling <[email protected]>
* fix: set the world state hasher to Poseidon2 * test: reactivate and migrate the tests for the accumulator * renames Bytes32ToHash -> Octuplet * test the consistency between hashLR and the normal "byte" hashing * fix: tailHash --------- Co-authored-by: Yao Galteland <[email protected]>
* bench: adds a standard benchmark with self-recursion * perf(poseidon2): optimize the poseidon2 compiler to create less columns and assign faster * fix: columns are not expressions by default * perf: optimize RS when rho is not 2 * clean the selfrecursion benchmark * fixup: reed-solomon * change the small field parameters * Update panic message in GetValBase method Signed-off-by: AlexandreBelling <[email protected]> * refactor: every 4 rounds * refactor: pack every 4 rounds together * pack rounds together, opt version * opt size * opt size * internalPackedSize = 4 * add one more parameter for constrain first term of sbox * best improvement * docs: key optimization * merge dev-small-fields * refactor: direct hashElement to save mem * update maxVal to reduce the mem by calling SetString * GC in commitment * directly return Poseidon2FieldHasherDigest * merge * opt: replace allkey computation to allkey len computation * opt: AllKeysAt--> AllKeysLenAt, avoid all key storage * optimal parameter * cleanup * merge * cleanup * opt: directly write a single elemt * remove StateStorer to save mem * fix: AllKeysLenAt output * address the comments * update a way to define maxVal to be more readable --------- Signed-off-by: AlexandreBelling <[email protected]> Co-authored-by: AlexandreBelling <[email protected]>
* merge * rm Write and ReadInt64On32Bytes, update to On16Bytes * refactor: WriteInt64On16Bytes and ReadInt64On16Bytes store 2 bytes in every 4 bytes(one field.Element) * add generic padding and remove padding helpers * refactor: read and write big int ok * refactor: ethadd + bytes32 * update: read and write int64 on 64 bytes * account ok * generate valid fieldhash * update 16 to 64 * accumulator ok, utils ok * add test vectors for account * add hash to ethaddress and fullBytes32 * add a new test vector for non dummy values * move test vectors together * move test vectors together * move the error check before RemovePadding * update byte count * use r.Read * update account size * docs: update test descriptions * update panic message * update ReadBigIntOn64Bytes panic message * update description, 32->64
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements issue(s) #
Checklist