Skip to content
Draft
Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
124 commits
Select commit Hold shift + click to select a range
62ca861
LeiosDemo202510: draft Leios mini protocols
nfrisby Oct 8, 2025
12e5667
Add the initial version of the demo script
dnadales Oct 10, 2025
7ebe7b5
Add node-0 to the path of different files
dnadales Oct 13, 2025
79d58e4
leiosdemo202510: add this demo exe
nfrisby Oct 13, 2025
97a4949
leiosdemo202510: combine via join
nfrisby Oct 13, 2025
1e71416
leiosdemo202510: suggestive check on MsgLeiosBlockTxsRequest
nfrisby Oct 13, 2025
f92cb19
Run a second Cardano node as downstream peer
dnadales Oct 14, 2025
35af3c9
leiosdemo202510: add msgLeiosBlockRequest and also stub limit on popC…
nfrisby Oct 14, 2025
f514ce0
leiosdemo202510: relocate SQL definitions near use
nfrisby Oct 14, 2025
cc15804
Scaffold leios-mvd NixOS test with leios-node
bladyjoker Oct 14, 2025
cbdd85a
Git ignore NixOS test history files and direnv
bladyjoker Oct 14, 2025
231a2a1
leiosdemo202510: avoid reversing long lists
nfrisby Oct 14, 2025
7efc564
leiosdemo202510: fallback for requests of many txs
nfrisby Oct 14, 2025
cc1e3b6
leiosdemo202510: avoid reversing lists 2
nfrisby Oct 14, 2025
ee4871b
leiosdemo202510: use map CBOR and handle MsgLeiosBlock
nfrisby Oct 14, 2025
9ad22ea
leiosdemo202510: handle MsgLeiosBlockTxs
nfrisby Oct 14, 2025
c4a9387
Implements immdb-node and Immutable DB NixOS systemd service
bladyjoker Oct 15, 2025
f78487b
Add `--address` flag to `immdb-server` executable
bladyjoker Oct 15, 2025
62db8e4
Add `address` options to `immdb-server` NixOS service
bladyjoker Oct 15, 2025
5b454c2
DNS tweaks to Leios MVD and formatting
bladyjoker Oct 15, 2025
7c25e7f
Add immutable db fixture (50 blocks)
bladyjoker Oct 15, 2025
ba949db
Configure the private testnet such that it doesn't update the system …
bladyjoker Oct 15, 2025
2cb6afa
Add README.md for Leios MVD and an actual test scenario condition
bladyjoker Oct 15, 2025
d2ab9f1
leiosdemo202510: merge ebBodies and ebClosures, use only points in CL…
nfrisby Oct 15, 2025
9e3d427
leiosdemo202510: minor refactor
nfrisby Oct 15, 2025
7036dd2
leiosdemo202510: add fetch-decision and hash-txs commands
nfrisby Oct 15, 2025
c9a6d68
leiosdemo202510: only use "point" in the table name ebPoints
nfrisby Oct 15, 2025
1c2a031
leiosdemo202510: add some of the missing Haddock
nfrisby Oct 15, 2025
41358c1
Add a script to analyze the two node log files
dnadales Oct 20, 2025
31c21cf
Hook the python script to the demo script
dnadales Oct 20, 2025
daaf2ea
Add requirements.txt for log_parser.py
dnadales Oct 20, 2025
2b7cbe9
Nixify the Leios demo environment
bladyjoker Oct 21, 2025
7bd63e1
Appliy black python formatter
bladyjoker Oct 21, 2025
a4d5b2a
Add Leios demo test data
bladyjoker Oct 21, 2025
c6d35dc
Nixify the Leios demo environment (#1724)
bladyjoker Oct 21, 2025
cbb0ae3
Add the IPython notebook for interactive analysis
bladyjoker Oct 21, 2025
0bd8498
Thread the onset of the reference slot through the immdb-server code
dnadales Oct 21, 2025
089702b
Pass a function that abstracts away slot delay calculation
dnadales Oct 21, 2025
0945ffd
Freeze the Python environment from Nix into requirements.txt
bladyjoker Oct 22, 2025
59587e8
leiosdemo202510: redo the fetch iteration logic and MsgLeiosBlockTxs …
nfrisby Oct 22, 2025
fb92641
leiosdemo202510: add the first bash script that mimics requests and r…
nfrisby Oct 22, 2025
ba40262
Implement mkGetSlotDelay
dnadales Oct 22, 2025
02dc37e
leiosdemo202510: add tx sharing to the generator
nfrisby Oct 22, 2025
1ce37c7
leiosdemo202510: fix off-by-one sterraf noticed
nfrisby Oct 22, 2025
dbd74db
Remove unnecessary venvShellHook
bladyjoker Oct 23, 2025
b543c22
Merge branch 'bladyjoker/leios-mvd-nixostests' into 1709-initial-demo…
bladyjoker Oct 23, 2025
893a4b6
Leios MVD setup with NixOS tests (#1718)
bladyjoker Oct 23, 2025
25879a4
Add new options for ImmDB NixOS service
bladyjoker Oct 23, 2025
464dfb0
leiosdemo202510: some renamings, add MkLeiosBlockRequest ctor
nfrisby Oct 23, 2025
88d080d
leiosdemo202510: some renamings, add EB body requests etc
nfrisby Oct 23, 2025
5eca7c5
leiosdemo202510: small polishing pass
nfrisby Oct 23, 2025
bdee13c
leiosdemo202510: also insert into TxCache
nfrisby Oct 23, 2025
e7e3104
leiosdemo202510: do not fetch txs already in TxCache
nfrisby Oct 23, 2025
3d5487e
leiosdemo202510: add cache-copy command
nfrisby Oct 24, 2025
ccbe693
leiosdemo202510: rename binder to nickname in myManifest.json
nfrisby Oct 24, 2025
2dafd96
Set all followert instructions to blocking
bladyjoker Oct 24, 2025
f69528b
Cosmetic changes to aid my own comprehension
bladyjoker Oct 24, 2025
f0e793f
Switch to using plotly for charting events
bladyjoker Oct 24, 2025
7b894e5
Merge remote-tracking branch 'origin/1709-initial-demo-setup' into nf…
nfrisby Oct 24, 2025
58abd61
leiosdemo202510: add mini protocol stubs to ImmDB server
nfrisby Oct 24, 2025
8ecc04b
Delay throwing ReachedImmutableTip to allow clients to fetch blocks
bladyjoker Oct 24, 2025
5976d2d
Adds journal logs from leios-node and adjust up the Python scripts
bladyjoker Oct 24, 2025
97784e7
leiosdemo202510: added generalized LeiosNotify peers to ouroboros-con…
nfrisby Oct 24, 2025
30f4ce8
Add arguments for the anchor slot and its corresponding onset
dnadales Oct 24, 2025
384f0fc
Add support for POSIX timestamps
dnadales Oct 24, 2025
31bffbe
Calculate slot onset time for each slot in the logs
dnadales Oct 24, 2025
45c59c7
Pass anchor slot and its onset to the log_parser script
dnadales Oct 24, 2025
31af74a
Add a sanity check for the slot onset calculation
dnadales Oct 24, 2025
d42ef8a
Make the script generate a scatter plot
dnadales Oct 24, 2025
2912259
WIP
nfrisby Oct 24, 2025
9571dbf
WIP the script
nfrisby Oct 24, 2025
4b3cc38
WIP my TODO list
nfrisby Oct 24, 2025
bf9731e
Revert "WIP the script"
nfrisby Oct 24, 2025
643f4b4
Merge remote-tracking branch 'origin/1709-initial-demo-setup' into nf…
nfrisby Oct 24, 2025
a8bd024
WIP hacking on script file
nfrisby Oct 25, 2025
64d8c2c
WIP fixup 291225921 LeiosSchedule parsing in immdb-server
nfrisby Oct 25, 2025
6c48bd8
leiosdemo202510: use in-mem table instead of batching retrievals
nfrisby Oct 25, 2025
350943e
leiosdemo202510: polishing model exe
nfrisby Oct 25, 2025
6c0dbbe
leiosdemo202510: adding Leios state to NodeKernel
nfrisby Oct 25, 2025
6cefae1
leiosdemo202510: switch from RealPoint to bespoke LeiosPoint
nfrisby Oct 25, 2025
2be294f
leiosdemo202510: add basic logic to LeiosNotify client
nfrisby Oct 25, 2025
6efd6d8
leiosdemo202510: migrate EbId logic from the model exe
nfrisby Oct 25, 2025
6f2a6af
fixup leiosdemo202510: LeiosNotify
nfrisby Oct 25, 2025
40cddd1
leiosdemo202510: add LeiosFetch to NodeToNode
nfrisby Oct 25, 2025
4238e12
fixup leiosdemo202510: LeiosNotify
nfrisby Oct 25, 2025
dc09aab
leiosdemo202510: add generalized LeiosFetch pipelined client
nfrisby Oct 25, 2025
15052f5
leiosdemo202510: switch requestsToSend to TVar, for convenience
nfrisby Oct 25, 2025
eff731f
leiosdemo202510: use MutVar instead of MVar for drainThePipe
nfrisby Oct 25, 2025
c3c90b6
leiosdemo202510: polishing demo script
nfrisby Oct 26, 2025
1c690a3
leiosdemo202510: add LeiosFetch to immdb-server
nfrisby Oct 26, 2025
f9dc066
leiosdemo202510: fixup LeiosNotify client
nfrisby Oct 26, 2025
6a25c0b
leiosdemo202510: add LeiosFetch server to node
nfrisby Oct 26, 2025
43b6bb8
leiosdemo202510: more direct LeiosNotify server
nfrisby Oct 26, 2025
9c5584f
leiosdemo202510: add fetch decision logic to node
nfrisby Oct 26, 2025
d8e11d7
leiosdemo202510: now sending LeiosFetch requests, but ignoring responses
nfrisby Oct 26, 2025
3a1e330
leiosdemo202510: update model exe EB codec to match
nfrisby Oct 26, 2025
4f7c88c
leiosdemo202510: now processing MsgLeiosBlock, but ignoring MsgLeiosB…
nfrisby Oct 26, 2025
8057562
WIP one minute copies
nfrisby Oct 27, 2025
c1a2c95
WIP bugfix, see longer message
nfrisby Oct 27, 2025
473d06f
WIP disable the most noisy traces
nfrisby Oct 27, 2025
6438535
WIP fixup warnings
nfrisby Oct 27, 2025
772f6ee
leiosdemo202510: drop the netstat stuff
nfrisby Oct 27, 2025
e7aed1e
leiosdemo202510: consolidate magic numbers (absorbing maxIngressQueue)
nfrisby Oct 27, 2025
fe3b41f
leiosdemo202510: consolidate ebId magic number
nfrisby Oct 27, 2025
c567c18
leiosdemo202510: also generate typical schedule.json file
nfrisby Oct 27, 2025
15c2a46
leiosdemo202510: track outstanding EbTxs count per EB
nfrisby Oct 27, 2025
4d7c1bc
leiosdemo202510: more interpretable packing of ebIds, for now
nfrisby Oct 27, 2025
b635ebd
leiosdemo202510: LeiosNotify server in node, but DB locked :/
nfrisby Oct 28, 2025
6e5d547
WIP HasCallStack for db
nfrisby Oct 28, 2025
41fbed6
WIP node-0 freezing???
nfrisby Oct 28, 2025
ff77ac3
WIP change host-addr?
nfrisby Oct 28, 2025
3973252
WIP fix missing finalize and dieStack = dieStack :'(
nfrisby Oct 28, 2025
19f5707
leiosdemo202510: LeiosFetch for downstream too
nfrisby Oct 28, 2025
ea74e1b
leiosdemo202510: add a 2nd node-under-test, but not seeing mixed traf…
nfrisby Oct 28, 2025
903890c
Revert "leiosdemo202510: add a 2nd node-under-test, but not seeing mi…
nfrisby Oct 28, 2025
1886752
leiosdemo202510: add missing VACUUM
nfrisby Oct 28, 2025
af08f6a
leiosdemo202510: do not elide CompletedBlockFetch
nfrisby Oct 28, 2025
1a50c3b
leiosdemo202510: hack leios-october-script.sh and log_parser.py to sh…
nfrisby Oct 28, 2025
49026a6
leiosdemo202510: disable scatter plot for now
nfrisby Oct 28, 2025
17b4292
leiosdemo202510: polishing the leios-october-script.sh
nfrisby Oct 28, 2025
c0d12b9
leiosdemo202510: add README-Leios-October-demo.md and demoManifest.json
nfrisby Oct 28, 2025
0720059
leiosdemo202510: add some documentation to README-Leios-October-demo.md
nfrisby Oct 28, 2025
660207b
leiosdemo202510: fix typos in README-Leios-October-demo.md
nfrisby Oct 28, 2025
1dba9a7
leiosdemo202510: update now that the simplest first experiment is wor…
nfrisby Oct 28, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions leiosdemo202510-first-script
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
#! /bin/bash
rm -f tmp/my.db tmp/up.db tmp/my.lfst tmp/demo tmp/reqs
set -eux
cabal build exe:leiosdemo202510
ln -s $(cabal list-bin exe:leiosdemo202510) tmp/demo
tmp/demo generate tmp/up.db tmp/my.lfst tmp/myManifest.json
cp tmp/{up,my}.db
sqlite3 tmp/my.db "SELECT ebSlot, PRINTF('%X', ebId), ebId FROM ebPoints ORDER BY ebId ASC" # dump points
sqlite3 tmp/my.db 'SELECT ebId, MAX(txOffset) FROM ebTxs GROUP BY ebId' # dump sizes
tmp/demo fetch-logic-iteration tmp/my.db tmp/my.lfst | tee -a tmp/reqs
sqlite3 tmp/my.db 'UPDATE ebTxs SET txBytes = NULL WHERE ebId = -9223372036843241473'
tmp/demo fetch-logic-iteration tmp/my.db tmp/my.lfst | tee -a tmp/reqs
tmp/demo MsgLeiosBlockTxsOffer tmp/my.lfst Alice -9223372036848484353 -9223372036843241473
tmp/demo fetch-logic-iteration tmp/my.db tmp/my.lfst | tee -a tmp/reqs
tmp/demo fetch-logic-iteration tmp/my.db tmp/my.lfst | tee -a tmp/reqs
tmp/demo MsgLeiosBlockTxsOffer tmp/my.lfst Bob -9223372036848484353 -9223372036843241473
tmp/demo fetch-logic-iteration tmp/my.db tmp/my.lfst | tee -a tmp/reqs
tmp/demo fetch-logic-iteration tmp/my.db tmp/my.lfst | tee -a tmp/reqs
sqlite3 tmp/my.db 'UPDATE ebTxs SET txBytes = NULL WHERE ebId = -9223372036848484353'
tmp/demo fetch-logic-iteration tmp/my.db tmp/my.lfst | tee -a tmp/reqs
tmp/demo fetch-logic-iteration tmp/my.db tmp/my.lfst | tee -a tmp/reqs
jq . tmp/my.lfst
sqlite3 tmp/my.db 'SELECT ebId FROM ebTxs WHERE txBytes IS NULL GROUP BY ebId' # dump incompletes
set +x
cat tmp/reqs | grep -e MSG | grep -e Alice | cut -d' ' -f3- | while IFS= read -r line; do
set -x
tmp/demo MsgLeiosBlockTxsRequest tmp/up.db $line | xxd -plain -revert >tmp/foo.bin
tmp/demo MsgLeiosBlockTxs tmp/my.db tmp/my.lfst Alice tmp/foo.bin
set +x
jq . tmp/my.lfst
sqlite3 tmp/my.db 'SELECT ebId FROM ebTxs WHERE txBytes IS NULL GROUP BY ebId' # dump incompletes
done
cat tmp/reqs | grep -e MSG | grep -e Bob | cut -d' ' -f3- | while IFS= read -r line; do
set -x
tmp/demo MsgLeiosBlockTxsRequest tmp/up.db $line | xxd -plain -revert >tmp/foo.bin
tmp/demo MsgLeiosBlockTxs tmp/my.db tmp/my.lfst Bob tmp/foo.bin
jq . tmp/my.lfst
sqlite3 tmp/my.db 'SELECT ebId FROM ebTxs WHERE txBytes IS NULL GROUP BY ebId' # dump incompletes
set +x
done
55 changes: 55 additions & 0 deletions myManifest.json

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,8 @@ import Ouroboros.Network.TxSubmission.Mempool.Reader
import Ouroboros.Network.TxSubmission.Outbound


import qualified Ouroboros.Network.Mux as ON

{-------------------------------------------------------------------------------
Handlers
-------------------------------------------------------------------------------}
Expand Down Expand Up @@ -838,6 +840,28 @@ initiator miniProtocolParameters version versionData Apps {..} =
})
version
versionData
<>
mempty { ON.withHot = ON.WithHot [
ON.MiniProtocol {
ON.miniProtocolNum = leiosNotifyMiniProtocolNum,
ON.miniProtocolStart = ON.StartOnDemand,
ON.miniProtocolLimits = ON.MiniProtocolLimits {
ON.maximumIngressQueue = addSafetyMargin $ 1 * 2 ^ (20 :: Int) -- 1 mebibyte
},
ON.miniProtocolRun = InitiatorProtocolOnly (MiniProtocolCb undefined)
}
, ON.MiniProtocol {
ON.miniProtocolNum = leiosFetchMiniProtocolNum,
ON.miniProtocolStart = ON.StartOnDemand,
ON.miniProtocolLimits = ON.MiniProtocolLimits {
ON.maximumIngressQueue = addSafetyMargin $ 50 * 2 ^ (20 :: Int) -- 50 mebibytes
},
ON.miniProtocolRun = InitiatorProtocolOnly (MiniProtocolCb undefined)
}
]
}
where
addSafetyMargin x = x + x `div` 10

-- | A bi-directional network application.
--
Expand Down Expand Up @@ -878,3 +902,32 @@ initiatorAndResponder miniProtocolParameters version versionData Apps {..} =
})
version
versionData
<>
mempty { ON.withHot = ON.WithHot [
ON.MiniProtocol {
ON.miniProtocolNum = leiosNotifyMiniProtocolNum,
ON.miniProtocolStart = ON.StartOnDemand,
ON.miniProtocolLimits = ON.MiniProtocolLimits {
ON.maximumIngressQueue = addSafetyMargin $ 1 * 2 ^ (20 :: Int) -- 1 mebibyte
},
ON.miniProtocolRun = InitiatorAndResponderProtocol (MiniProtocolCb undefined) (MiniProtocolCb undefined)
}
, ON.MiniProtocol {
ON.miniProtocolNum = leiosFetchMiniProtocolNum,
ON.miniProtocolStart = ON.StartOnDemand,
ON.miniProtocolLimits = ON.MiniProtocolLimits {
ON.maximumIngressQueue = addSafetyMargin $ 50 * 2 ^ (20 :: Int) -- 50 mebibytes
},
ON.miniProtocolRun = InitiatorAndResponderProtocol (MiniProtocolCb undefined) (MiniProtocolCb undefined)
}
]
}
where
addSafetyMargin x = x + x `div` 10


leiosNotifyMiniProtocolNum :: MiniProtocolNum
leiosNotifyMiniProtocolNum = MiniProtocolNum 18

leiosFetchMiniProtocolNum :: MiniProtocolNum
leiosFetchMiniProtocolNum = MiniProtocolNum 19
Loading
Loading