Drop support for Python 3.7 and fix CI (#2400)

IvoDD · web-flow · commit 3f236ac4da42 · 2025-06-13T11:11:44.000+01:00
#### Reference Issues/PRs  #### What does this implement or fix? Drops support for end of life Python 3.7. And removes codepaths checking for `python < 3.8` Also tweaks clang docker image to fix CI issue (where cmake was finding wrong python version). Also had to install cmake3 because now by default the manylinux_2_28 images have cmake4. #### Any other comments? Link to a [successful build](https://github.com/man-group/ArcticDB/actions/runs/15613614393) using this branch's workflow and `arcticdb-dev-clang:fixed-path` docker image which I built locally. And a link to [successful build with analysis tools](https://github.com/man-group/ArcticDB/actions/runs/15613127117/). Fails asv checks due to network error but benchmarking works fine #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details>
diff --git a/.github/workflows/README.md b/.github/workflows/README.md
@@ -149,7 +149,7 @@ This logic is used in [build.yml](build.yml).
 <tr><th>inputs.cibw_image_tag</th><td>Linux only. As built by cibw_docker_image.yml workflow</td>
 <tr><th>inputs.cibw_version</th><td>Follower only. Must match the cibw_image_tag</td>
 <tr><th>inputs.python_deps_ids</th><td>Follower test matrix parameter. JSON string.</td>
-<tr><th>inputs.python3</th><td>Specifies the Python 3 minor version (e.g. 6, 7, 8, etc.)</td>
+<tr><th>inputs.python3</th><td>Specifies the Python 3 minor version (e.g. 8, 9, 10, etc.)</td>
 </table>
 
 ## [tag.yml](tag.yml)
diff --git a/.github/workflows/analysis_workflow.yml b/.github/workflows/analysis_workflow.yml
@@ -3,6 +3,7 @@ on:
   workflow_dispatch:
     inputs:
       run_all_benchmarks:
+        description: Run all benchmarks
         type: boolean
         default: false
       dev_image_tag:
@@ -125,7 +126,7 @@ jobs:
     name: Executes asv tests checks
     timeout-minutes: 120
     runs-on: ubuntu-latest
-    container: ghcr.io/man-group/arcticdb-dev:${{ inputs.dev_image_tag }}
+    container: ghcr.io/man-group/arcticdb-dev:${{ inputs.dev_image_tag || 'latest' }}
     env:
         SCCACHE_GHA_VERSION: ${{vars.SCCACHE_GHA_VERSION || 1}} # Setting this env var enables the caching
         VCPKG_NUGET_USER: ${{secrets.VCPKG_NUGET_USER || github.repository_owner}}
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
@@ -145,7 +145,8 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python3: ${{fromJson(vars.LINUX_PYTHON_VERSIONS || '[6, 7, 8, 9, 10, 11]')}}
+        # This uses old arcticdb versions which won't work on python < 3.12
+        python3: ${{fromJson(vars.LINUX_PYTHON_VERSIONS || '[8, 9, 10, 11]')}}
         arcticdb_version: ["oldest", "latest"]
         include:
           - python_deps_ids: [""]
@@ -168,7 +169,8 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python3: ${{fromJson(vars.WINDOWS_PYTHON_VERSIONS || '[7, 8, 9, 10, 11]')}}
+        # This uses old arcticdb versions which won't work on python < 3.12
+        python3: ${{fromJson(vars.WINDOWS_PYTHON_VERSIONS || '[8, 9, 10, 11]')}}
         arcticdb_version: ["oldest", "latest"]
         include:
           - python_deps_ids: [""]
@@ -202,7 +204,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python3: ${{fromJson(vars.LINUX_PYTHON_VERSIONS || '[7, 8, 9, 10, 11, 12, 13]')}}
+        python3: ${{fromJson(vars.LINUX_PYTHON_VERSIONS || '[8, 9, 10, 11, 12, 13]')}}
         include:
           - python_deps_ids: [""]
             matrix_override: ${{fromJson(needs.common_config.outputs.linux_matrix)}}
@@ -253,7 +255,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python3: ${{fromJson(vars.WINDOWS_PYTHON_VERSIONS || '[7, 8, 9, 10, 11, 12, 13]')}}
+        python3: ${{fromJson(vars.WINDOWS_PYTHON_VERSIONS || '[8, 9, 10, 11, 12, 13]')}}
     name: 3.${{matrix.python3}} Windows
     uses: ./.github/workflows/build_steps.yml
     secrets: inherit
@@ -277,7 +279,8 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python3: ${{fromJson(vars.LINUX_PYTHON_VERSIONS || '[6, 7, 8, 9, 10, 11]')}}
+        # This uses old arcticdb versions which won't work on python < 3.12
+        python3: ${{fromJson(vars.LINUX_PYTHON_VERSIONS || '[8, 9, 10, 11]')}}
         arcticdb_version: ["oldest", "latest"]
         include:
           - python_deps_ids: [""]
@@ -299,7 +302,8 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python3: ${{fromJson(vars.WINDOWS_PYTHON_VERSIONS || '[7, 8, 9, 10, 11]')}}
+        # This uses old arcticdb versions which won't work on python < 3.12
+        python3: ${{fromJson(vars.WINDOWS_PYTHON_VERSIONS || '[8, 9, 10, 11]')}}
         arcticdb_version: ["oldest", "latest"]
         include:
           - python_deps_ids: [""]
diff --git a/.github/workflows/installation_tests.yml b/.github/workflows/installation_tests.yml
@@ -64,9 +64,6 @@ jobs:
         # This way we cover all python versions, OS-es and installation repos 
         # in just 8 combinations
         include:
-          - os: ubuntu-22.04
-            python: "3.7"
-            use_conda: "no"
           - os: windows-latest
             python: "3.8"
             use_conda: "no"
diff --git a/README.md b/README.md
@@ -45,7 +45,7 @@ ArcticDB is designed from the outset to be resilient; there is no single point o
 
 ### Prebuilt binary availability
 
-|                       | PyPI (Python 3.7 - 3.13) | conda-forge (Python 3.9 - 3.13) |
+|                       | PyPI (Python 3.8 - 3.13) | conda-forge (Python 3.9 - 3.13) |
 | --------------------- | - | - |
 | Linux (Intel/AMD)     | ✔️ | ✔️ |
 | Windows (Intel/AMD)   | ✔️ | ➖ |
diff --git a/docker/Dockerfile b/docker/Dockerfile
@@ -3,7 +3,7 @@ FROM quay.io/pypa/manylinux_2_28_x86_64
 RUN dnf update -y
 RUN dnf remove -y 'gcc-toolset-*'
 RUN dnf install -y zip flex bison gcc-toolset-11 gcc-toolset-11-gdb gcc-toolset-11-libatomic-devel krb5-devel cyrus-sasl-devel openssl-devel \
-unzip tar epel-release jq wget libcurl-devel git-lfs \
+unzip tar epel-release jq wget libcurl-devel git-lfs cmake3 \
 python3.11-devel python3.11-pip perl-IPC-Cmd
 
 RUN dnf groupinstall -y 'Development Tools'
@@ -23,4 +23,4 @@ ENV CMAKE_C_COMPILER=/opt/rh/gcc-toolset-11/root/bin/gcc
 ENV CXX=/opt/rh/gcc-toolset-11/root/bin/g++
 ENV CMAKE_CXX_COMPILER=/opt/rh/gcc-toolset-11/root/bin/g++
 ENV LD_LIBRARY_PATH=/opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst
-ENV PATH=/opt/rh/gcc-toolset-11/root/usr/bin:/opt/python/cp311-cp311/bin:${PATH}
+ENV PATH=/opt/rh/gcc-toolset-11/root/usr/bin:/opt/python/cp311-cp311/bin:/usr/bin:${PATH}
diff --git a/docker/Dockerfile_clang b/docker/Dockerfile_clang
@@ -2,9 +2,9 @@ FROM quay.io/pypa/manylinux_2_28_x86_64
 
 RUN dnf update -y
 RUN dnf remove -y 'gcc-toolset-*'
-# Looks like clang depends on the gcc-toolset-13, so try to install the libatomic-devel from there
-RUN dnf install -y zip flex bison clang llvm lldb gcc-toolset-13-libatomic-devel krb5-devel cyrus-sasl-devel openssl-devel \
-unzip tar epel-release jq wget libcurl-devel git-lfs \
+# Looks like clang depends on the gcc-toolset-14, so try to install the libatomic-devel from there
+RUN dnf install -y zip flex bison clang llvm lldb gcc-toolset-14-libatomic-devel krb5-devel cyrus-sasl-devel openssl-devel \
+unzip tar epel-release jq wget libcurl-devel git-lfs cmake3 \
 python3.11-devel python3.11-pip perl-IPC-Cmd
 
 RUN dnf groupinstall -y 'Development Tools'
@@ -24,4 +24,4 @@ ENV CMAKE_C_COMPILER=/usr/bin/clang
 ENV CXX=/usr/bin/clang++
 ENV CMAKE_CXX_COMPILER=/usr/bin/clang++
 ENV LD_LIBRARY_PATH=/usr/lib64:/usr/lib
-ENV PATH=/usr/bin:/opt/python/cp311-cp311/bin:${PATH}
+ENV PATH=/opt/python/cp311-cp311/bin:/usr/bin:${PATH}
diff --git a/docs/mkdocs/docs/index.md b/docs/mkdocs/docs/index.md
@@ -32,7 +32,7 @@ This section will cover installation, setup and basic usage. More details on bas
 
 ### Installation
 
-ArcticDB supports Python 3.7 - 3.13.
+ArcticDB supports Python 3.8 - 3.13.
 
 To install, simply run:
 
diff --git a/pyproject.toml b/pyproject.toml
@@ -30,7 +30,7 @@ line-length = 120
 
 [tool.black]
 line-length = 120
-target_version = ['py36', 'py37', 'py38', 'py39', 'py310', 'py311']
+target_version = ['py38', 'py39', 'py310', 'py311', 'py312', 'py313']
 preview = true
 # This must be kept in sync with the version in setup.cfg.
 exclude = '''
diff --git a/python/arcticdb/storage_fixtures/s3.py b/python/arcticdb/storage_fixtures/s3.py
@@ -874,10 +874,7 @@ def cleanup_bucket(self, b: S3Bucket):
             try:
                 self._s3_admin.delete_bucket(Bucket=b.bucket)
             except botocore.exceptions.ClientError as e:
-                # There is a problem with xdist on Windows 3.7
-                # where we try to clean up the bucket but it's already gone
-                is_win_37 = platform.system() == "Windows" and sys.version_info[:2] == (3, 7)
-                if e.response["Error"]["Code"] != "NoSuchBucket" and not is_win_37:
+                if e.response["Error"]["Code"] != "NoSuchBucket":
                     raise e
         else:
             try:
diff --git a/python/arcticdb/util/utils.py b/python/arcticdb/util/utils.py
@@ -14,11 +14,7 @@
 import time
 import sys
 from typing import Dict 
-if sys.version_info >= (3, 8): 
-    from typing import Literal, Any, List, Tuple, Union, get_args
-else: 
-    from typing_extensions import Literal
-    from typing import Any, List, Tuple, Union
+from typing import Literal, Any, List, Tuple, Union, get_args
 import numpy as np
 import pandas as pd
 
@@ -30,14 +26,9 @@
 ArcticIntType = Union[np.uint8, np.uint16, np.uint32, np.uint64, np.int8, np.int16, np.int32, np.int64]
 ArcticFloatType = Union[np.float64, np.float32]
 ArcticTypes = Union[ArcticIntType, ArcticFloatType, str]
-if sys.version_info >= (3, 8):
-    supported_int_types_list = list(get_args(ArcticIntType))
-    supported_float_types_list = list(get_args(ArcticFloatType))
-    supported_types_list = list(get_args(ArcticTypes))
-else:
-    supported_int_types_list = [np.uint8, np.uint16, np.uint32, np.uint64, np.int8, np.int16, np.int32, np.int64]
-    supported_float_types_list = [np.float64, np.float32]
-    supported_types_list = [str] + supported_int_types_list + supported_float_types_list
+supported_int_types_list = list(get_args(ArcticIntType))
+supported_float_types_list = list(get_args(ArcticFloatType))
+supported_types_list = list(get_args(ArcticTypes))
 
 
 class GitHubSanitizingHandler(logging.StreamHandler):
diff --git a/python/arcticdb/version_store/processing.py b/python/arcticdb/version_store/processing.py
@@ -5,6 +5,7 @@
 
 As of the Change Date specified in that file, in accordance with the Business Source License, use of this software will be governed by the Apache License, version 2.0.
 """
+from __future__ import annotations
 from collections import namedtuple
 import copy
 from dataclasses import dataclass
@@ -867,9 +868,7 @@ def resample(
         self._python_clauses = self._python_clauses + [PythonResampleClause(rule=rule, closed=boundary_map[closed], label=boundary_map[label], offset=offset_ns, origin=origin)]
         return self
 
-    # TODO: specify type of other must be QueryBuilder with from __future__ import annotations once only Python 3.7+
-    # supported
-    def then(self, other):
+    def then(self, other : QueryBuilder):
         """
         Applies processing specified in other after any processing already defined for this QueryBuilder.
 
@@ -887,9 +886,7 @@ def then(self, other):
         self._python_clauses = self._python_clauses + other._python_clauses
         return self
 
-    # TODO: specify type of other must be QueryBuilder with from __future__ import annotations once only Python 3.7+
-    # supported
-    def prepend(self, other):
+    def prepend(self, other : QueryBuilder):
         """
         Applies processing specified in other before any processing already defined for this QueryBuilder.
 
diff --git a/python/tests/unit/arcticdb/version_store/test_normalization.py b/python/tests/unit/arcticdb/version_store/test_normalization.py
@@ -274,8 +274,6 @@ def test_write_tz(lmdb_version_store, sym, tz):
             # We convert all timezones to pytz, so we expect to convert to a pytz timezone.
             expected_df.index = expected_df.index.tz_convert(pytz.timezone(str(tz)))
     assert_frame_equal(expected_df, result)
-    if tz == du.tz.gettz("UTC") and sys.version_info < (3, 7):
-        pytest.skip("Timezone files don't seem to have ever worked properly on Python 3.6")
     start_ts, end_ts = lmdb_version_store.get_timerange_for_symbol(sym)
     assert isinstance(start_ts, datetime.datetime)
     assert isinstance(end_ts, datetime.datetime)
diff --git a/python/tests/util/mark.py b/python/tests/util/mark.py
@@ -117,7 +117,7 @@
 FORK_SUPPORTED = pytest.mark.skipif(WINDOWS, reason="Fork not supported on Windows")
 
 ## MEMRAY supports linux and macos and python 3.8 and above
-MEMRAY_SUPPORTED = (sys.version_info >= (3, 8)) and (MACOS or LINUX)
+MEMRAY_SUPPORTED = (MACOS or LINUX)
 MEMRAY_TESTS_MARK = pytest.mark.skipif(
     not MEMRAY_SUPPORTED, reason="MEMRAY supports linux and macos and python 3.8 and above"
 )
@@ -140,8 +140,7 @@
 
 PANDAS_2_COMPAT_TESTS_MARK = pytest.mark.skipif(
     MACOS_CONDA_BUILD
-    or sys.version.startswith("3.7")  # Pandas 2 not available in 3.7 or 3.8
-    or sys.version.startswith("3.8")
+    or sys.version.startswith("3.8") # Pandas 2 not available in 3.8
     or sys.version.startswith("3.12")
     or sys.version.startswith("3.13"),  # Waiting for https://github.com/man-group/ArcticDB/issues/2008
     reason="Skipping compatibility tests because macOS conda builds don't have an available PyPi arcticdb version and "
diff --git a/setup.cfg b/setup.cfg
@@ -37,7 +37,6 @@ install_requires =
     numpy 
     pandas
     attrs
-    dataclasses ; python_version < '3.7'
     protobuf >=3.5.0.post1, < 6 # Per https://github.com/grpc/grpc/blob/v1.45.3/requirements.txt
     msgpack >=0.5.0 # msgpack 0.5.0 is required for strict_types argument, needed for correct pickling fallback
     pyyaml
@@ -46,7 +45,7 @@ install_requires =
 [flake8]
 # max line length for black
 max-line-length = 120
-target-version = ['py37']
+target-version = ['py38']
 # Default flake8 3.5 ignored flags
 ignore=
     # check ignored by default in flake8. Meaning unclear.
@@ -128,8 +127,8 @@ Testing =
     pymongo
     trustme
     psutil
-    memray; python_version >= '3.7' and (platform_system == 'Linux' or platform_system == 'Darwin')
-    pytest-memray; python_version >= '3.8'and (platform_system == 'Linux' or platform_system == 'Darwin')
+    memray; platform_system == 'Linux' or platform_system == 'Darwin'
+    pytest-memray; platform_system == 'Linux' or platform_system == 'Darwin'
 
 [options.entry_points]
 console_scripts =
diff --git a/setup.py b/setup.py
@@ -65,14 +65,6 @@ def run(self):
                 # No available on PyPI for this configuration.
                 # See the last release's: https://pypi.org/project/protobuf/3.20.3/#files
                 print(f"Python protobuf {proto_ver} is not officially supported on Python {python}. Skipping...")
-            elif not ARCTICDB_USING_CONDA and python <= (3, 6) and proto_ver >= "4":
-                # No available on PyPI for this configuration
-                # See the first release's: https://pypi.org/project/protobuf/4.21.1/#files
-                print(f"Python protobuf {proto_ver} do not run on Python {python}. Skipping...")
-            elif not ARCTICDB_USING_CONDA and python <= (3, 7) and proto_ver >= "5":
-                # Not compatible with Python<=3.7 as
-                # GRPCIO >= 1.64.0 is not available on Python<=3.7 https://pypi.org/project/grpcio-tools/1.64.0/#files
-                print(f"Python protobuf {proto_ver} does not run on Python {python}. Skipping...")
             elif not ARCTICDB_USING_CONDA and python >= (3, 13) and proto_ver < "5":
                 # Not compatible with protobuf<5.26.1 as
                 # Python 3.13 requires GRPCIO >= 1.66.2, which requires protobuf >=5.26.1 https://github.com/grpc/grpc/blob/6fa8043bf9befb070b846993b59a3348248e6566/requirements.txt#L4
