Skip to content

Commit bc314fe

Browse files
jam-duna traces upd: Reverse trie key bit ordering (#10)
Based on w3f/jamtestvectors#14 Also: * jam-duna: assurances first trace: just 1 epoch * add note re bootstrap services, adjust codec => bin in README
1 parent d6a30f5 commit bc314fe

File tree

425 files changed

+12970
-11170
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

425 files changed

+12970
-11170
lines changed

README.md

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -46,20 +46,21 @@ Each validator communicates with other nodes using JAMNP's QUIC protocol. Depend
4646

4747
| `mode` | `safrole` | `assurance` | `finality` | `conformance` |
4848
|---------------|-----------|-------------|------------|----------------|
49-
| QUIC / JAM Codec | Block, Ticket | WorkPackage, Guarantee, Assurance, EC TBD | TBD: Announcement, Vote, ... | TBD: Dispute |
5049
| Tickets: E_T | x | x | x | x |
5150
| Guarantees: E_G | | x | x | x |
5251
| Assurances: E_A | | x | x | x |
5352
| Preimages: E_P | | x | x | x |
5453
| Refine/Accumulate PVM | | x | x | x |
54+
| Bootstrap Services: Assign | | x | x | x |
5555
| Audit/Announcements | | | x | x |
5656
| GRANDPA | | | x | x |
5757
| Disputes: E_D | | | | x |
5858
| BLS | | | | x |
5959
| BEEFY | | | | x |
6060
| Authorization Pool/Queue | | | | x |
61-
| Privileged Services | | | | x |
61+
| Bootstrap Services: Other | | | | x |
6262
| State | C4, C6, C7, C8, C9, C11, C13 | C10, C12 | - | C1, C2, C3, C5 |
63+
| JAMSNP | CE128,129,131 | + CE133-135,137,141 | + CE142,143,144,145 | + CE132 |
6364
| Timeline | Q4 2024 | Q1 2025 | early Q2 2025 | late Q2 2025 |
6465

6566
We aim for 5-10 teams to successfully establish their own working testnets in Q4, with collaborative efforts beginning around sub0@Devcon7+JAM0 in mid-November.
@@ -83,17 +84,15 @@ directory.
8384

8485
If you have service code (in any language, including privileged services) suitable for `mode=assurances`, put it in `services/${team}/${servicename}` (e.g. `services/strongly-web3/fib`). See [services](./services) for examples in progress.
8586

86-
To model submission of work packages utilizing a service, traces can include the receipt by the first validator, e.g in `traces/${mode}/${team}/workpackages/${core}.{codec,json}$`
87-
8887
## JAM Implementer Submissions
8988

90-
Building a collaborative JAM Testnet requires teams share Docker image URLs, Traces, and Testnet configurations, but not code.
89+
Building a collaborative JAM Testnet requires teams share Traces (supporting validation independent of JAMSNP) and Docker image URLs (requiring JAMSNP), but not code.
9190

9291
To contribute:
9392

94-
- Submit a PR to add a Docker image URL below.
95-
- Add your fully working `docker-compose.yml` to `testnet/${mode}/${team}` (e.g. `testnet/safrole/jam-duna/docker-compose.yml`)
96-
- Add your sample trace to `traces/${mode}/${team}` (e.g. `traces/safrole/jam-duna/data`)
93+
- Submit a PR to add your traces and/or a Docker image URL below.
94+
- For traces of blocks and states, add your sample trace to `traces/${mode}/${team}` (e.g. `traces/safrole/jam-duna`)
95+
- For Docker image urls, add below and a fully working `docker-compose.yml` to `testnet/${mode}/${team}` (e.g. `testnet/safrole/jam-duna/docker-compose.yml`)
9796

9897
| team | Docker Image URL |
9998
|---------------|--------------------------------------------------------|
@@ -130,3 +129,5 @@ To contribute:
130129
| subjam | TBD |
131130

132131
For simplicity, use lowercase for `${team}` and `${mode}`. Dashes and underscores are ok, spaces/punctations are not.
132+
133+
Everyone must follow JAM Prize rules and can politely refuse collaboration with any team for any reason.

services/README.md

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,9 @@
44

55
JAM Services consist of PVM Byte code placed on-chain with 4 entry points.
66
To support the testing of these JAM Services, JAM Implementation teams are encouraged to place
7-
JAM Service code, in any language here.
7+
JAM Service code, in any language here. Note that JAM Service code is not necessary for teams to
8+
collaborate on JAM testnet, but it may support useful debugging. Services may include a
9+
_bootstrap service_ included in genesis state.
810

911
* Raw assembly, with [polkatool](https://github.com/koute/polkavm/tree/master/tools/polkatool)
1012
* Rust, see [polkatool](https://github.com/koute/polkavm/tree/master/tools/polkatool)
@@ -21,10 +23,13 @@ Each JAM Service should have raw source code, build instructions, and a _JAM-rea
2123
* `accumulate` (entry point 10)
2224
* `on_transfer` (entry point 15)
2325

24-
Furthermore, as privileged services themselves are needed to place
25-
code blobs on-chain and create JAM Services utilizing these code
26-
blobs, it is critical that teams have a "generic" privileged service
27-
to have a JAM Testnet in "assurances" mode. This is an open question.
26+
Furthermore, a _bootstrap service_ is needed in genesis state to support many different privileged operations:
27+
* create JAM Services
28+
* assigning coretime
29+
* requesting preimages
30+
* changing the validator keysets
31+
32+
These always-accumulate/privileged services use the same JAMSNP work package submission+sharing processes as non-privileged service. Teams are encouraged to share Bootstrap service code along with their genesis configs.
2833

2934
## Raw PVM Assembly code
3035

traces/README.md

Lines changed: 47 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -7,53 +7,72 @@ each others state transitions (including state roots) and check each others JAM
77

88
If your team has a sequence of Blocks + States, please submit a PR going into this directory, containing at minimum easily parsed / decoded files for each `epoch` and $phase`
99

10-
* Blocks: `$epoch_$phase.json` `$epoch_$phase.codec`
11-
* States: `$epoch_$phase.json` `$epoch_$phase.codec`
10+
* Blocks: `$epoch_$phase.json` `$epoch_$phase.bin`
11+
* States: `$epoch_$phase.json` `$epoch_$phase.bin`
1212

1313
The combination of JSON + codec is useful for debugging purposes.
1414

1515
Here is a sample output:
1616
```
17-
# ls safrole/$TEAM_NAME/blocks/
18-
0_0.codec 0_11.json 0_5.codec 0_8.json 1_10.codec 1_3.json 1_7.codec 2_0.json 2_2.codec 2_5.json 2_9.codec 3_10.json 3_4.codec 3_7.json
19-
0_0.json 0_2.codec 0_5.json 0_9.codec 1_10.json 1_4.codec 1_7.json 2_1.codec 2_2.json 2_6.codec 2_9.json 3_11.codec 3_4.json 3_8.codec
20-
0_1.codec 0_2.json 0_6.codec 0_9.json 1_11.codec 1_4.json 1_8.codec 2_1.json 2_3.codec 2_6.json 3_0.codec 3_11.json 3_5.codec 3_8.json
21-
0_1.json 0_3.codec 0_6.json 1_0.codec 1_11.json 1_5.codec 1_8.json 2_10.codec 2_3.json 2_7.codec 3_0.json 3_2.codec 3_5.json 3_9.codec
22-
0_10.codec 0_3.json 0_7.codec 1_0.json 1_2.codec 1_5.json 1_9.codec 2_10.json 2_4.codec 2_7.json 3_1.codec 3_2.json 3_6.codec 3_9.json
23-
0_10.json 0_4.codec 0_7.json 1_1.codec 1_2.json 1_6.codec 1_9.json 2_11.codec 2_4.json 2_8.codec 3_1.json 3_3.codec 3_6.json 4_0.codec
24-
0_11.codec 0_4.json 0_8.codec 1_1.json 1_3.codec 1_6.json 2_0.codec 2_11.json 2_5.codec 2_8.json 3_10.codec 3_3.json 3_7.codec 4_0.json
25-
26-
# ls safrole/$TEAM_NAME/state_snapshots/
27-
0_0.codec 0_11.json 0_5.codec 0_8.json 1_10.codec 1_3.json 1_7.codec 2_0.json 2_2.codec 2_5.json 2_9.codec 3_10.json 3_4.codec 3_7.json
28-
0_0.json 0_2.codec 0_5.json 0_9.codec 1_10.json 1_4.codec 1_7.json 2_1.codec 2_2.json 2_6.codec 2_9.json 3_11.codec 3_4.json 3_8.codec
29-
0_1.codec 0_2.json 0_6.codec 0_9.json 1_11.codec 1_4.json 1_8.codec 2_1.json 2_3.codec 2_6.json 3_0.codec 3_11.json 3_5.codec 3_8.json
30-
0_1.json 0_3.codec 0_6.json 1_0.codec 1_11.json 1_5.codec 1_8.json 2_10.codec 2_3.json 2_7.codec 3_0.json 3_2.codec 3_5.json 3_9.codec
31-
0_10.codec 0_3.json 0_7.codec 1_0.json 1_2.codec 1_5.json 1_9.codec 2_10.json 2_4.codec 2_7.json 3_1.codec 3_2.json 3_6.codec 3_9.json
32-
0_10.json 0_4.codec 0_7.json 1_1.codec 1_2.json 1_6.codec 1_9.json 2_11.codec 2_4.json 2_8.codec 3_1.json 3_3.codec 3_6.json 4_0.codec
33-
0_11.codec 0_4.json 0_8.codec 1_1.json 1_3.codec 1_6.json 2_0.codec 2_11.json 2_5.codec 2_8.json 3_10.codec 3_3.json 3_7.codec 4_0.json
17+
# ls safrole/$TEAM_NAME/blocks
18+
349445_0.bin 349445_2.bin 349445_6.bin 349446_0.bin 349446_2.bin 349446_6.bin 349447_0.bin 349447_2.bin 349447_6.bin 349448_0.bin 349448_2.bin 349448_6.bin 349449_0.bin
19+
349445_0.json 349445_2.json 349445_6.json 349446_0.json 349446_2.json 349446_6.json 349447_0.json 349447_2.json 349447_6.json 349448_0.json 349448_2.json 349448_6.json 349449_0.json
20+
349445_1.bin 349445_3.bin 349445_7.bin 349446_1.bin 349446_3.bin 349446_7.bin 349447_1.bin 349447_3.bin 349447_7.bin 349448_1.bin 349448_3.bin 349448_7.bin 349449_1.bin
21+
349445_1.json 349445_3.json 349445_7.json 349446_1.json 349446_3.json 349446_7.json 349447_1.json 349447_3.json 349447_7.json 349448_1.json 349448_3.json 349448_7.json 349449_1.json
22+
349445_10.bin 349445_4.bin 349445_8.bin 349446_10.bin 349446_4.bin 349446_8.bin 349447_10.bin 349447_4.bin 349447_8.bin 349448_10.bin 349448_4.bin 349448_8.bin
23+
349445_10.json 349445_4.json 349445_8.json 349446_10.json 349446_4.json 349446_8.json 349447_10.json 349447_4.json 349447_8.json 349448_10.json 349448_4.json 349448_8.json
24+
349445_11.bin 349445_5.bin 349445_9.bin 349446_11.bin 349446_5.bin 349446_9.bin 349447_11.bin 349447_5.bin 349447_9.bin 349448_11.bin 349448_5.bin 349448_9.bin
25+
349445_11.json 349445_5.json 349445_9.json 349446_11.json 349446_5.json 349446_9.json 349447_11.json 349447_5.json 349447_9.json 349448_11.json 349448_5.json 349448_9.json
26+
# ls safrole/$TEAM_NAME/state_snapshots
27+
349445_0.bin 349445_2.bin 349445_6.bin 349446_0.bin 349446_2.bin 349446_6.bin 349447_0.bin 349447_2.bin 349447_6.bin 349448_0.bin 349448_2.bin 349448_6.bin 349449_0.bin
28+
349445_0.json 349445_2.json 349445_6.json 349446_0.json 349446_2.json 349446_6.json 349447_0.json 349447_2.json 349447_6.json 349448_0.json 349448_2.json 349448_6.json 349449_0.json
29+
349445_1.bin 349445_3.bin 349445_7.bin 349446_1.bin 349446_3.bin 349446_7.bin 349447_1.bin 349447_3.bin 349447_7.bin 349448_1.bin 349448_3.bin 349448_7.bin 349449_1.bin
30+
349445_1.json 349445_3.json 349445_7.json 349446_1.json 349446_3.json 349446_7.json 349447_1.json 349447_3.json 349447_7.json 349448_1.json 349448_3.json 349448_7.json 349449_1.json
31+
349445_10.bin 349445_4.bin 349445_8.bin 349446_10.bin 349446_4.bin 349446_8.bin 349447_10.bin 349447_4.bin 349447_8.bin 349448_10.bin 349448_4.bin 349448_8.bin
32+
349445_10.json 349445_4.json 349445_8.json 349446_10.json 349446_4.json 349446_8.json 349447_10.json 349447_4.json 349447_8.json 349448_10.json 349448_4.json 349448_8.json
33+
349445_11.bin 349445_5.bin 349445_9.bin 349446_11.bin 349446_5.bin 349446_9.bin 349447_11.bin 349447_5.bin 349447_9.bin 349448_11.bin 349448_5.bin 349448_9.bin
34+
349445_11.json 349445_5.json 349445_9.json 349446_11.json 349446_5.json 349446_9.json 349447_11.json 349447_5.json 349447_9.json 349448_11.json 349448_5.json 349448_9.json
3435
```
3536

3637
Teams should be able to take a directory of `$mode/$TEAM_NAME/blocks` and:
3738
- read a genesis state `genesis.json` for the `$mode`
38-
- read blocks/*.codec and decode all blocks successfully
39+
- read blocks/*.bin and decode all blocks successfully with the JAM Codec
3940
- validate each block, checking the stateroot and block hash at a minimum
4041

41-
Note: For JSON, the intention is for JSON object attribute names to be as
42-
close as possible to w3f `codec` test vectors. However, different
43-
teams are likely to choose slightly different JSON attribute names for
44-
some state outside the test vectors, so validation between the two
45-
teams should be based on the JAM codec files instead of the JSON
46-
content. The JSON should be used solely for supporting conversations
47-
and ease of debugging.
42+
Here is a sample run of a `validatetraces` that does the above:
43+
```
44+
# ./validatetraces ~/jam-duna/jamtestnet/traces/assurances/jam_duna
45+
VALIDATED Block 349462_0.bin => State 349462_0.bin | [N3] H_t=4193544 H_r=0910..b69d EpochMarker(η1=151f..3ead)
46+
VALIDATED Block 349462_1.bin => State 349462_1.bin | [N3] H_t=4193545 H_r=27d0..5869 |E_T|=3
47+
VALIDATED Block 349462_2.bin => State 349462_2.bin | [N2] H_t=4193546 H_r=6413..540d |E_T|=3
48+
VALIDATED Block 349462_3.bin => State 349462_3.bin | [N1] H_t=4193547 H_r=134b..f686 |E_T|=3 |E_G|=1
49+
VALIDATED Block 349462_4.bin => State 349462_4.bin | [N4] H_t=4193548 H_r=4329..66d0 |E_T|=3 |E_A|=5
50+
VALIDATED Block 349462_5.bin => State 349462_5.bin | [N2] H_t=4193549 H_r=5c46..e964 |E_T|=3
51+
VALIDATED Block 349462_6.bin => State 349462_6.bin | [N1] H_t=4193550 H_r=4aaa..94e2 |E_T|=3
52+
VALIDATED Block 349462_7.bin => State 349462_7.bin | [N1] H_t=4193551 H_r=9f96..ceb4
53+
VALIDATED Block 349462_8.bin => State 349462_8.bin | [N1] H_t=4193552 H_r=3e82..7b05
54+
VALIDATED Block 349462_9.bin => State 349462_9.bin | [N3] H_t=4193553 H_r=5838..d51c
55+
VALIDATED Block 349462_10.bin => State 349462_10.bin | [N3] H_t=4193554 H_r=f300..38c6 WinningTickets(12)
56+
VALIDATED Block 349462_11.bin => State 349462_11.bin | [N5] H_t=4193555 H_r=111d..3ca5
57+
Trace validation completed successfully.
58+
```
4859

60+
Note that JAM codec versions are used, not JSON files. JSON files are
61+
provided for debugging only. In particular, JSON object attribute
62+
names to be as close as possible to w3f `codec` test vectors.
63+
However, different teams are likely to choose slightly different JSON
64+
attribute names for some state variables outside the test vectors, so
65+
validation between two teams should be based on the JAM codec files
66+
instead of the JSON content.
4967

5068
## Modes
5169

5270
* For mode `safrole`, only the first 4 epochs are useful or needed following this [JAM Safrole Model](https://docs.google.com/spreadsheets/d/1ueAisCMOx7B-m_fXMLT0FXBxfVzydJyr-udE8jKwDN8/edit?gid=615049643#gid=615049643)
5371

5472
* For mode `assurance`, `finality`, and `conformance` additional epochs may be required using sample work packages.
5573

74+
* Additional modes can be added between based on JAM implementer community interest.
75+
5676
## Notes
5777

5878
This is a PoC at this point and any team submission should not be taken as authoritative in any way.
59-

0 commit comments

Comments
 (0)