Change timing of AH/RC pre and post snapshots (#247)

* Change timing of AH/RC pre and post snapshots

Pre-migration:
  - RC snapshot: taken at block with state transition to AccountsMigrationInit
  - AH snapshot: taken at block with state transition to DataMigrationOngoing + 1
Post-migration:
  - RC snapshot: taken at block with state transition to CoolOff
  - AH snapshot: taken at block with state transition to CoolOff

Driven-by: update runtime to include CoolOff as state transition for AH.

* Cleanup

* Cleanup

* More cleanup

* Fix AH pre-snapshot block detection

* Update zombie-bite-scripts/helpers.ts

Co-authored-by: Javier Viola <[email protected]>

* Fix cool-off check

* Raise timeout for RC snapshot to 12h to accommoadate for Polkadot migration

* Update runtime to point to runtime muharem-ahm-polkadot

* Suppress try-runtime verbose output to avoid child_process exec maxBuffer

* Take all 4 snapshots past AH and RC CoolOff

* monitoring: poll via RPC instead of WS long-live connection

* incorporate ci changes/debug

* show output in ci

* debug self hosted with ops

* fix get block numbers

* small nits for ci

* fix path to check

* fix tmpl

* fix tmpl

* clean in ubuntu

* Fixes for use post-db as starting point (#251)

* test

* test ci without docker

* allow to use a post-db

* add option to workflow

* add option to workflow

* fix option for action

* fix action

* workaround db snap

* workaround db snap

* workaround db snap

* change zombie-bite install

* changes to run ts test with post db

* add script

* Improved zombie-bite termination logic (#250)

* changed zombie-bite stop logic

* refactor

* Bump PET to include PAHM E2E test suites (#249)

* Bump PET to include PAHM E2E test suites

* Default network to polkadot in PET workflow

* Bump PET submodule to commit with passing PAH tests

It also includes snapshots for passing PAH E2E tests, from October
21st.

* Bump PET to include PAHM child bounty tests

\also has KAH vesting test fixes, which are also replicated in PAHM.

---------

Co-authored-by: Andrii <[email protected]>
Co-authored-by: Alexandre R. Baldé <[email protected]>

* more work on ci

* multiple patterns for download pre-db

* add support for old pre-db

* typo and less interval in CI

---------

Co-authored-by: Javier Viola <[email protected]>
Co-authored-by: Javier Viola <[email protected]>
Co-authored-by: Andrii <[email protected]>
Co-authored-by: Alexandre R. Baldé <[email protected]>
Co-authored-by: muharem <[email protected]>

Author: 5 people

.github/actions/download-artifact/action.yml

@@ -35,9 +35,13 @@ runs:
         echo "downloading artifacts from https://github.com/paritytech/ahm-dryrun/actions/runs/${RUN_ID}"
 
         if [[ $NAME_PATTERN == "" ]]; then
-          gh run download $RUN_ID -p "${NAME_PATTERN}" -D $TMP_DIR
+          gh run download $RUN_ID -D $TMP_DIR
         else
-          gh run download $RUN_ID -p "${NAME_PATTERN}" -D $TMP_DIR
+          PATTERNS=""
+          for p in $(echo $NAME_PATTERN| tr "," "\n");do
+            PATTERNS="$PATTERNS -p $p "
+          done
+          gh run download $RUN_ID $PATTERNS -D $TMP_DIR
         fi;
 
         ARTIFACTS_DIR=$(ls -t $TMP_DIR|head -1)

.github/actions/download-doppelganger-binaries/action.yml

@@ -26,7 +26,7 @@ runs:
           VERSION_TO_DOWNLOAD=$(curl https://api.github.com/repos/paritytech/doppelganger-wrapper/releases/latest | jq -r .tag_name)
         fi;
         for bin in doppelganger doppelganger-parachain polkadot-execute-worker polkadot-prepare-worker; do
-          echo "downloading $bin";
+          echo "downloading $bin from https://github.com/paritytech/doppelganger-wrapper/releases/download/$VERSION_TO_DOWNLOAD/$bin";
           curl -L -o $DESTINATION_PATH/$bin https://github.com/paritytech/doppelganger-wrapper/releases/download/$VERSION_TO_DOWNLOAD/$bin;
           chmod 755 $DESTINATION_PATH/$bin;
         done;

.github/actions/download-post-migration-db/action.yml

@@ -21,12 +21,31 @@ runs:
       run: |
         set -euo pipefail
         TMP_DIR="$(mktemp -d)"
-        
-        curl -sSLf -H "Authorization: Bearer ${GITHUB_TOKEN}" -o "${TMP_DIR}/artifact.zip" "${ARTIFACT_URL}"
-        
+
+        FILENAME=$(basename ${ARTIFACT_URL})
+
+        IS_GITHUB=$(echo ${ARTIFACT_URL}| grep -o github || true)
+        IS_ZIP=$(echo ${ARTIFACT_URL}| grep -o zip || true)
+        if [[ $IS_GITHUB == "github" ]];then
+          curl -sSLf -H "Authorization: Bearer ${GITHUB_TOKEN}" -o "${TMP_DIR}/${FILENAME}" "${ARTIFACT_URL}"
+        else
+          curl -sSLf -o "${TMP_DIR}/${FILENAME}" "${ARTIFACT_URL}"
+        fi;
+
         rm -rf "${DESTINATION_PATH}"
         mkdir -p "${DESTINATION_PATH}"
-        unzip -q "${TMP_DIR}/artifact.zip" -d "${DESTINATION_PATH}"
+
+        if [[ $IS_ZIP == "zip" ]];then
+          unzip -q "${TMP_DIR}/${FILENAME}" -d "${DESTINATION_PATH}"
+        else
+          tar -xvf "${TMP_DIR}/${FILENAME}" -C "${DESTINATION_PATH}"
+        fi;
+
         rm -rf "${TMP_DIR}"
-        
-        ls "${DESTINATION_PATH}"
+        echo "ls dst path"
+        ls "${DESTINATION_PATH}"
+        mv ${DESTINATION_PATH}/pahm/spawn ${DESTINATION_PATH}/
+        mv ${DESTINATION_PATH}/pahm/ready.json ${DESTINATION_PATH}/
+        mv ${DESTINATION_PATH}/pahm/ports.json ${DESTINATION_PATH}/
+        echo "ls dst path"
+        ls "${DESTINATION_PATH}"

.github/actions/wait-zb-network-ready/action.yml

@@ -17,7 +17,7 @@ runs:
       shell: bash
       run: |
           echo "Waiting for 'network is up and running' message..."
-          timeout 300 bash -c '
+          timeout 900 bash -c '
             while ! grep -q "network is up and running" $LOG_FILE 2>/dev/null; do
               echo "Still waiting for network... last log line:"
               tail -1 $LOG_FILE

.github/workflows/migration-schedule-check.yml

@@ -23,7 +23,7 @@ permissions: {}
 jobs:
   get-run-id:
     runs-on: ubuntu-latest
-    if: ${{ (github.event_name == 'workflow_dispatch' && github.event.inputs.run-id == '') || 'true' }}
+    if: ${{ (github.event_name == 'workflow_dispatch' && github.event.inputs.run-id == '') }}
     outputs:
       RUN_ID: ${{ steps.get_run_id.outputs.RUN_ID }}
     steps:

.github/workflows/rust-tests-common.yml

@@ -56,38 +56,28 @@ jobs:
           echo "downloading artifacts from https://github.com/paritytech/ahm-dryrun/actions/runs/${RUN_ID}"
           gh run download $RUN_ID -p "${NETWORK}-try-runtime*" -D $SNAPS_PATH
           ls -ltr $SNAPS_PATH
-          ARTIFACTS_DIR_PRE=$(ls -t $SNAPS_PATH|grep try-runtime-snaps-pre)
-          ARTIFACTS_DIR_POST=$(ls -t $SNAPS_PATH|grep try-runtime-snaps-post)
-          echo "ARTIFACTS_DIR_PRE=$SNAPS_PATH/$ARTIFACTS_DIR_PRE" >> $GITHUB_OUTPUT
-          echo "ARTIFACTS_DIR_POST=$SNAPS_PATH/$ARTIFACTS_DIR_POST" >> $GITHUB_OUTPUT
+          # Single artifact contains all 4 snapshots
+          ARTIFACTS_DIR=$(ls -t $SNAPS_PATH|grep try-runtime-snaps|grep -v pre|grep -v post|head -1)
+          echo "ARTIFACTS_DIR=$SNAPS_PATH/$ARTIFACTS_DIR" >> $GITHUB_OUTPUT
 
       - name: verify_and_move_snaps
         shell: bash
         env:
           NETWORK: ${{ inputs.network }}
         run: |
-          for i in rc-pre.snap ah-pre.snap;do
-            if [ ! -f "${{ steps.download_artifacts.outputs.ARTIFACTS_DIR_PRE }}/${NETWORK}-${i}" ]; then
+          # All 4 snapshots are in a single artifact directory
+          for i in rc-pre.snap ah-pre.snap rc-post.snap ah-post.snap;do
+            if [ ! -f "${{ steps.download_artifacts.outputs.ARTIFACTS_DIR }}/${NETWORK}-${i}" ]; then
               echo "ERROR: Could not find ${NETWORK}-${i}"
               exit 1
             else
-              mv "${{ steps.download_artifacts.outputs.ARTIFACTS_DIR_PRE }}/${NETWORK}-${i}" $SNAPS_PATH
-            fi
-          done;
-
-          for i in rc-post.snap ah-post.snap;do
-            if [ ! -f "${{ steps.download_artifacts.outputs.ARTIFACTS_DIR_POST }}/${NETWORK}-${i}" ]; then
-              echo "ERROR: Could not find ${NETWORK}-${i}"
-              exit 1
-            else
-              mv "${{ steps.download_artifacts.outputs.ARTIFACTS_DIR_POST }}/${NETWORK}-${i}" $SNAPS_PATH
+              mv "${{ steps.download_artifacts.outputs.ARTIFACTS_DIR }}/${NETWORK}-${i}" $SNAPS_PATH
             fi
           done;
 
           echo "final files in ${SNAPS_PATH}"
           ls -ltr $SNAPS_PATH
 
-
       - name: run_rust_tests
         shell: bash
         env: