Merge branch 'master' into take-fix

Author: lucascolley

.flake8

@@ -0,0 +1,2 @@
+[flake8]
+select = F

.github/workflows/lint.yml

@@ -6,15 +6,11 @@ jobs:
   build:
 
     runs-on: ubuntu-latest
-    strategy:
-      matrix:
-        python-version: [3.8, 3.9]
-
     steps:
-    - uses: actions/checkout@v1
+    - uses: actions/checkout@v4
     - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v1
+      uses: actions/setup-python@v5
       with:
-        python-version: ${{ matrix.python-version }}
+        python-version: "3.10"
     - name: Run pre-commit hook
-      uses: pre-commit/action@v2.0.3
+      uses: pre-commit/action@v3.0.1

.github/workflows/numpy.yml renamed to .github/workflows/test.yml

@@ -1,4 +1,4 @@
-name: NumPy Array API
+name: Test Array API Strict
 
 on: [push, pull_request]
 
@@ -22,10 +22,13 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        python -m pip install numpy==1.26.2
+        python -m pip install array-api-strict
         python -m pip install -r requirements.txt
     - name: Run the test suite
       env:
-        ARRAY_API_TESTS_MODULE: numpy.array_api
+        ARRAY_API_TESTS_MODULE: array_api_strict
+        ARRAY_API_STRICT_API_VERSION: 2024.12
       run: |
-        pytest -v -rxXfE --ci --skips-file numpy-skips.txt
+        pytest -v -rxXfE --skips-file array-api-strict-skips.txt array_api_tests/
+        # We also have internal tests that isn't really necessary for adopters
+        pytest -v -rxXfE meta_tests/

README.md

@@ -3,8 +3,8 @@
 This is the test suite for array libraries adopting the [Python Array API
 standard](https://data-apis.org/array-api/latest).
 
-Note the suite is still a **work in progress**. Feedback and contributions are
-welcome!
+Keeping full coverage of the spec is an on-going priority as the Array API evolves.
+Feedback and contributions are welcome!
 
 ## Quickstart
 
@@ -33,11 +33,29 @@ You need to specify the array library to test. It can be specified via the
 `ARRAY_API_TESTS_MODULE` environment variable, e.g.
 
 ```bash
-$ export ARRAY_API_TESTS_MODULE=numpy.array_api
+$ export ARRAY_API_TESTS_MODULE=array_api_strict
+```
+
+To specify a runtime-defined module, define `xp` using the `exec('...')` syntax:
+
+```bash
+$ export ARRAY_API_TESTS_MODULE="exec('import quantity_array, numpy; xp = quantity_array.quantity_namespace(numpy)')"
 ```
 
 Alternately, import/define the `xp` variable in `array_api_tests/__init__.py`.
 
+### Specifying the API version
+
+You can specify the API version to use when testing via the
+`ARRAY_API_TESTS_VERSION` environment variable, e.g.
+
+```bash
+$ export ARRAY_API_TESTS_VERSION="2023.12"
+```
+
+Currently this defaults to the array module's `__array_api_version__` value, and
+if that attribute doesn't exist then we fallback to `"2021.12"`.
+
 ### Run the suite
 
 Simply run `pytest` against the `array_api_tests/` folder to run the full suite.
@@ -138,9 +156,9 @@ issues](https://github.com/data-apis/array-api-tests/issues/) to us.
 
 ## Running on CI
 
-See our existing [GitHub Actions workflow for
-Numpy](https://github.com/data-apis/array-api-tests/blob/master/.github/workflows/numpy.yml)
-for an example of using the test suite on CI.
+See our existing [GitHub Actions workflow for `array-api-strict`](https://github.com/data-apis/array-api-tests/blob/master/.github/workflows/test.yml)
+for an example of using the test suite on CI. Note [`array-api-strict`](https://github.com/data-apis/array-api-strict)
+is an implementation of the array API that uses NumPy under the hood.
 
 ### Releases
 
@@ -154,19 +172,6 @@ library to fail.
 
 ### Configuration
 
-#### API version
-
-You can specify the API version to use when testing via the
-`ARRAY_API_TESTS_VERSION` environment variable. Currently this defaults to the
-array module's `__array_api_version__` value, and if that attribute doesn't
-exist then we fallback to `"2021.12"`.
-
-#### CI flag
-
-Use the `--ci` flag to run only the primary and special cases tests. You can
-ignore the other test cases as they are redundant for the purposes of checking
-compliance.
-
 #### Data-dependent shapes
 
 Use the `--disable-data-dependent-shapes` flag to skip testing functions which have
@@ -260,7 +265,7 @@ jobs:
 > There are several ways to avoid this problem:
 >
 > - Increase the maximum number of examples, e.g., by adding `--max-examples
->   200` to the test command (the default is `100`, see below). This will
+>   200` to the test command (the default is `20`, see below). This will
 >   make it more likely that the failing case will be found, but it will also
 >   make the tests take longer to run.
 > - Don't use `-o xfail_strict=True`. This will make it so that if an XFAIL
@@ -281,11 +286,44 @@ jobs:
 The tests make heavy use
 [Hypothesis](https://hypothesis.readthedocs.io/en/latest/). You can configure
 how many examples are generated using the `--max-examples` flag, which
-defaults to `100`. Lower values can be useful for quick checks, and larger
+defaults to `20`. Lower values can be useful for quick checks, and larger
 values should result in more rigorous runs. For example, `--max-examples
 10_000` may find bugs where default runs don't but will take much longer to
 run.
 
+#### Skipping Dtypes
+
+The test suite will automatically skip testing of inessential dtypes if they
+are not present on the array module namespace, but dtypes can also be skipped
+manually by setting the environment variable `ARRAY_API_TESTS_SKIP_DTYPES` to
+a comma separated list of dtypes to skip. For example
+
+```
+ARRAY_API_TESTS_SKIP_DTYPES=uint16,uint32,uint64 pytest array_api_tests/
+```
+
+Note that skipping certain essential dtypes such as `bool` and the default
+floating-point dtype is not supported.
+
+#### Turning xfails into skips
+
+Keeping a large number of ``xfails`` can have drastic effects on the run time. This is due
+to the way `hypothesis` works: when it detects a failure, it does a large amount
+of work to simplify the failing example.
+If the run time of the test suite becomes a problem, you can use the
+``ARRAY_API_TESTS_XFAIL_MARK`` environment variable: setting it to ``skip`` skips the
+entries from the ``xfail.txt`` file instead of xfailing them. Anecdotally, we saw
+speed-ups by a factor of 4-5---which allowed us to use 4-5 larger values of
+``--max-examples`` within the same time budget.
+
+#### Limiting the array sizes
+
+The test suite generates random arrays as inputs to functions it tests. "unvectorized"
+tests iterate over elements of arrays, which might be slow. If the run time becomes
+a problem, you can limit the maximum number of elements in generated arrays by
+setting the environment variable ``ARRAY_API_TESTS_MAX_ARRAY_SIZE`` to the
+desired value. By default, it is set to 1024.
+
 
 ## Contributing
 
@@ -355,26 +393,6 @@ into a release. If you want, you can add release notes, which GitHub can
 generate for you.
 
 
-## Future plans
-
-Keeping full coverage of the spec is an on-going priority as the Array API
-evolves.
-
-Additionally, we have features and general improvements planned. Work on such
-functionality is guided primarily by the concerete needs of developers
-implementing and using the Array API—be sure to [let us
-know](https://github.com/data-apis/array-api-tests/issues) any limitations you
-come across.
-
-* A dependency graph for every test case, which could be used to modify pytest's
-  collection so that low-dependency tests are run first, and tests with faulty
-  dependencies would skip/xfail.
-
-* In some tests we've found it difficult to find appropaite assertion parameters
-  for output values (particularly epsilons for floating-point outputs), so we
-  need to review these and either implement assertions or properly note the lack
-  thereof.
-
 ---
 
 <sup>1</sup>The only exceptions to having just one primary test per function are:

array-api

@@ -1,37 +1,13 @@
-# copy not implemented
-array_api_tests/test_creation_functions.py::test_asarray_arrays
-# https://github.com/numpy/numpy/issues/20870
-array_api_tests/test_data_type_functions.py::test_can_cast
-# The return dtype for trace is not consistent in the spec
-# https://github.com/data-apis/array-api/issues/202#issuecomment-952529197
-array_api_tests/test_linalg.py::test_trace
-# waiting on NumPy to allow/revert distinct NaNs for np.unique
-# https://github.com/numpy/numpy/issues/20326#issuecomment-1012380448
-array_api_tests/test_set_functions.py
-
-# newaxis not included in numpy namespace as of v1.26.2
-array_api_tests/test_constants.py::test_newaxis
-
-# linalg.solve issue in numpy.array_api as of v1.26.2 (see numpy#25146)
-array_api_tests/test_linalg.py::test_solve
-# numpy.array_api needs updating... or replaced on CI
-array_api_tests/test_linalg.py::test_cross
-
-# https://github.com/numpy/numpy/issues/21373
-array_api_tests/test_array_object.py::test_getitem
-
-# missing copy arg
-array_api_tests/test_signatures.py::test_func_signature[reshape]
-
-# does not (yet) raise an exception for zero-dimensional inputs to nonzero
-array_api_tests/test_searching_functions.py::test_nonzero_zerodim_error
-
-# https://github.com/numpy/numpy/issues/21211
-array_api_tests/test_special_cases.py::test_iop[__iadd__(x1_i is -0 and x2_i is -0) -> -0]
+# Known special case issue in NumPy. Not worth working around here
 # https://github.com/numpy/numpy/issues/21213
 array_api_tests/test_special_cases.py::test_iop[__ipow__(x1_i is -infinity and x2_i > 0 and not (x2_i.is_integer() and x2_i % 2 == 1)) -> +infinity]
 array_api_tests/test_special_cases.py::test_iop[__ipow__(x1_i is -0 and x2_i > 0 and not (x2_i.is_integer() and x2_i % 2 == 1)) -> +0]
-# noted diversions from spec
+
+# The test suite is incorrectly checking sums that have loss of significance
+# (https://github.com/data-apis/array-api-tests/issues/168)
+array_api_tests/test_statistical_functions.py::test_sum
+
+# Stubs have a comment: (**note**: libraries may return ``NaN`` to match Python behavior.); Apparently, all libraries do just that
 array_api_tests/test_special_cases.py::test_binary[floor_divide(x1_i is +infinity and isfinite(x2_i) and x2_i > 0) -> +infinity]
 array_api_tests/test_special_cases.py::test_binary[floor_divide(x1_i is +infinity and isfinite(x2_i) and x2_i < 0) -> -infinity]
 array_api_tests/test_special_cases.py::test_binary[floor_divide(x1_i is -infinity and isfinite(x2_i) and x2_i > 0) -> -infinity]
@@ -50,3 +26,9 @@ array_api_tests/test_special_cases.py::test_iop[__ifloordiv__(x1_i is -infinity
 array_api_tests/test_special_cases.py::test_iop[__ifloordiv__(x1_i is -infinity and isfinite(x2_i) and x2_i < 0) -> +infinity]
 array_api_tests/test_special_cases.py::test_iop[__ifloordiv__(isfinite(x1_i) and x1_i > 0 and x2_i is -infinity) -> -0]
 array_api_tests/test_special_cases.py::test_iop[__ifloordiv__(isfinite(x1_i) and x1_i < 0 and x2_i is +infinity) -> -0]
+
+# FIXME needs array-api-strict >=2.3.2
+array_api_tests/test_data_type_functions.py::test_finfo
+array_api_tests/test_data_type_functions.py::test_finfo_dtype
+array_api_tests/test_data_type_functions.py::test_iinfo
+array_api_tests/test_data_type_functions.py::test_iinfo_dtype

numpy-skips.txt renamed to array-api-strict-skips.txt

@@ -11,28 +11,47 @@
 
 
 # You can comment the following out and instead import the specific array module
-# you want to test, e.g. `import numpy.array_api as xp`.
+# you want to test, e.g. `import array_api_strict as xp`.
 if "ARRAY_API_TESTS_MODULE" in os.environ:
-    xp_name = os.environ["ARRAY_API_TESTS_MODULE"]
-    _module, _sub = xp_name, None
-    if "." in xp_name:
-        _module, _sub = xp_name.split(".", 1)
-    xp = import_module(_module)
-    if _sub:
-        try:
-            xp = getattr(xp, _sub)
-        except AttributeError:
-            # _sub may be a submodule that needs to be imported. WE can't
-            # do this in every case because some array modules are not
-            # submodules that can be imported (like mxnet.nd).
-            xp = import_module(xp_name)
+    env_var = os.environ["ARRAY_API_TESTS_MODULE"]
+    if env_var.startswith("exec('") and env_var.endswith("')"):
+        script = env_var[6:][:-2]
+        namespace = {}
+        exec(script, namespace)
+        xp = namespace["xp"]
+        xp_name = xp.__name__
+    else:
+        xp_name = env_var
+        _module, _sub = xp_name, None
+        if "." in xp_name:
+            _module, _sub = xp_name.split(".", 1)
+        xp = import_module(_module)
+        if _sub:
+            try:
+                xp = getattr(xp, _sub)
+            except AttributeError:
+                # _sub may be a submodule that needs to be imported. We can't
+                # do this in every case because some array modules are not
+                # submodules that can be imported (like mxnet.nd).
+                xp = import_module(xp_name)
 else:
     raise RuntimeError(
         "No array module specified - either edit __init__.py or set the "
         "ARRAY_API_TESTS_MODULE environment variable."
     )
 
 
+# If xp.bool is not available, like in some versions of NumPy and CuPy, try
+# patching in xp.bool_.
+try:
+    xp.bool
+except AttributeError as e:
+    if hasattr(xp, "bool_"):
+        xp.bool = xp.bool_
+    else:
+        raise e
+
+
 # We monkey patch floats() to always disable subnormals as they are out-of-scope
 
 _floats = st.floats
@@ -67,7 +86,7 @@ def _from_dtype(*a, **kw):
 
 
 api_version = os.getenv(
-    "ARRAY_API_TESTS_VERSION", getattr(xp, "__array_api_version__", "2021.12")
+    "ARRAY_API_TESTS_VERSION", getattr(xp, "__array_api_version__", "2024.12")
 )
 xps = array_api.make_strategies_namespace(xp, api_version=api_version)
 

array_api_tests/__init__.py

@@ -1,5 +1,5 @@
 from ._array_module import (isnan, all, any, equal, not_equal, logical_and,
-                            logical_or, isfinite, greater, less, less_equal,
+                            logical_or, isfinite, greater, less_equal,
                             zeros, ones, full, bool, int8, int16, int32,
                             int64, uint8, uint16, uint32, uint64, float32,
                             float64, nan, inf, pi, remainder, divide, isinf,
@@ -164,6 +164,16 @@ def notequal(x, y):
 
     return not_equal(x, y)
 
+def less(x, y):
+    """
+    Same as less(x, y) except it allows comparing uint64 with signed int dtypes
+    """
+    if x.dtype == uint64 and dh.dtype_signed[y.dtype]:
+        return xp.where(y < 0, xp.asarray(False), xp.less(x, xp.astype(y, uint64)))
+    if y.dtype == uint64 and dh.dtype_signed[x.dtype]:
+        return xp.where(x < 0, xp.asarray(True), xp.less(xp.astype(x, uint64), y))
+    return xp.less(x, y)
+
 def assert_exactly_equal(x, y, msg_extra=None):
     """
     Test that the arrays x and y are exactly equal.