Skip to content

Commit 5c22644

Browse files
marcofavoritobimarcofavoritobi
committed
ci: downgrade macos image to 11
This change was needed because of a nondeterminism in how the macos-12 (=macos-latest at the time of writing) runner image was instantiated by GitHub Actions. In particular, for runner version 2.311.0, the operating system was nondeterministically chosen between 12.6.9 and 12.7.1, as it can be seen in these two GH runs: - https://github.com/bancaditalia/black-it/actions/runs/7128997647/job/19412070642 - https://github.com/bancaditalia/black-it/actions/runs/7128997647/job/19412070868 This is a known issue in the community: see actions/runner-images#8642 Furthermore, at the time of writing, it is suspected that Python code that uses pytables is not stable over macos at version 12.7.1 (but this problem does not occur on v12.6.9!), as witnessed by this issue: PyTables/PyTables#1093 What happened in our CI workflow is that tests resulted in a flaky outcome, but not because of our tests, but due to the nondeterminism in how the macos runner was instantiated by GitHub Actions. This was not our unique attempt. In fact, we first tried to improve the preliminary configuration steps for the testing environment, as follows: - if: matrix.os == 'macos-latest' name: Install dependencies (macos-latest) run: | # for some reason this seems to be a necessary step for MacOS images, but not for Ubuntu and Windows brew install hdf5 brew install c-blosc # https://stackoverflow.com/questions/73029883/could-not-find-hdf5-installation-for-pytables-on-m1-mac echo "HDF5_DIR=/usr/local/" >> $GITHUB_ENV echo "BLOSC_DIR=/usr/local/" >> $GITHUB_ENV This attempt consisted in installing HDF5 and C-blosc explicitly in the preliminary steps on CI for MacOS images. By the way, this step was part of the installation instructions of PyTables v3.9.2: https://github.com/PyTables/PyTables/blob/v3.9.2/README.rst#installation , although it seems this step is not actually needed. Moreover, as explained here: https://stackoverflow.com/questions/73029883/could-not-find-hdf5-installation-for-pytables-on-m1-mac, sometimes must be explicitly set the environment path variables where HDF5 ahd C-blosc are installed For some reason this is not needed on Ubuntu and Windows runners. However, despite our efforts in this direction, the issue with pytables persisted. From the logs of the CI: (https://github.com/bancaditalia/black-it/actions/runs/7121188681/job/19389914846?pr=78#step:5:62) ``` Fatal Python error: Illegal instruction Current thread 0x000000011783c600 (most recent call first): File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/node.py", line 246 in __init__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/leaf.py", line 264 in __init__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/carray.py", line 200 in __init__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/earray.py", line 143 in __init__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/file.py", line 1345 in create_earray File "/Users/runner/work/black-it/black-it/black_it/utils/json_pandas_checkpointing.py", line 211 in save_calibrator_state File "/Users/runner/work/black-it/black-it/tests/test_utils/test_pandas_json_checkpointing.py", line 58 in test_save_and_load_calibrator_state File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/python.py", line 1792 in runtest File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 262 in <lambda> File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 341 in from_call File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 261 in call_runtest_hook File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 222 in call_and_report File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 133 in runtestprotocol File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/main.py", line 350 in pytest_runtestloop File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/main.py", line 325 in _main File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/main.py", line 271 in wrap_session File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/config/__init__.py", line 169 in main File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/config/__init__.py", line 192 in console_main File "/Users/runner/work/black-it/black-it/.tox/py310/bin/pytest", line 8 in <module> ``` Therefore, the current workaround we adopted was to downgrade the version of the macos runner to the previous version, 11.
1 parent 17aea3d commit 5c22644

File tree

1 file changed

+15
-5
lines changed

1 file changed

+15
-5
lines changed

‎.github/workflows/test.yml

+15-5
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,8 @@ jobs:
1313

1414
strategy:
1515
matrix:
16-
os: [ubuntu-latest, macos-latest, windows-latest]
16+
# on macos-11 instead of macos-latest, see https://github.com/bancaditalia/black-it/issues/79
17+
os: [ubuntu-latest, macos-11, windows-latest]
1718
python-version: [3.8, 3.9, "3.10", "3.11"]
1819

1920
timeout-minutes: 30
@@ -37,14 +38,23 @@ jobs:
3738
# env:
3839
# ACTIONS_ALLOW_UNSECURE_COMMANDS: true
3940
# run: ...
40-
- if: ${{ (matrix.os == 'ubuntu-latest') || (matrix.os == 'macos-latest') }}
41-
name: Unit tests and coverage (ubuntu-latest, macos-latest)
41+
- if: matrix.os == 'macos-11'
42+
name: Install dependencies (macos-11)
43+
run: |
44+
# for some reason this seems to be a necessary step for MacOS images, but not for Ubuntu and Windows
45+
brew install hdf5
46+
brew install c-blosc
47+
# https://stackoverflow.com/questions/73029883/could-not-find-hdf5-installation-for-pytables-on-m1-mac
48+
echo "HDF5_DIR=/usr/local/" >> $GITHUB_ENV
49+
echo "BLOSC_DIR=/usr/local/" >> $GITHUB_ENV
50+
- if: ${{ (matrix.os == 'ubuntu-latest') || (matrix.os == 'macos-11') }}
51+
name: Unit tests and coverage (ubuntu-latest, macos-11)
4252
run: |
4353
pyver=`echo ${{ matrix.python-version }} | tr -d "."`
4454
tox -e py${pyver} -- -m 'not e2e'
4555
tox -e py${pyver}-nb
46-
- if: ${{ (matrix.os == 'ubuntu-latest') || (matrix.os == 'macos-latest') }}
47-
name: Notebook tests (ubuntu-latest, macos-latest)
56+
- if: ${{ (matrix.os == 'ubuntu-latest') || (matrix.os == 'macos-11') }}
57+
name: Notebook tests (ubuntu-latest, macos-11)
4858
run: |
4959
pyver=`echo ${{ matrix.python-version }} | tr -d "."`
5060
tox -e py${pyver}-nb

0 commit comments

Comments
 (0)