Is your feature request related to a problem? Please describe.
Centrosome's Zernike calculation ends up allocating several complex arrays of image size x 30 (the number of polynomials we calculate), which means for some not-too-huge images (5120^2), we can end up running out of memory to run Zernike calculation even on a 64GB-memory machine. (See example stack trace below).
I'd need to do a deep dive on that code to see if we can in any way sparsify the array, if we NEED the complex data type, if we can maybe calculate each of the 30 polynomials independently, and/or calculate each object in only its neighborhood independently; I imagine at least one if not many of these might be doable. My guess as to why it wasn't already is that it would probably cause a net slowdown on smaller images, but for large image support it might be worth it (and/or we could consider a code path to split based on image size).
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/cellprofiler_core/pipeline/_pipeline.py", line 976, in run_with_yield
self.run_module(module, workspace)
File "/usr/local/lib/python3.8/dist-packages/cellprofiler_core/pipeline/_pipeline.py", line 1298, in run_module
module.run(workspace)
File "/usr/local/src/CellProfiler/cellprofiler/modules/measureobjectsizeshape.py", line 524, in run
self.run_on_objects(object_name, workspace)
File "/usr/local/src/CellProfiler/cellprofiler/modules/measureobjectsizeshape.py", line 586, in run_on_objects
features_to_record = self.analyze_objects(objects, desired_properties)
File "/usr/local/src/CellProfiler/cellprofiler/modules/measureobjectsizeshape.py", line 652, in analyze_objects
zf_l = centrosome.zernike.zernike(
File "/usr/local/lib/python3.8/dist-packages/centrosome/zernike.py", line 195, in zernike
zf = construct_zernike_polynomials(x, y, zernike_indexes, mask)
File "/usr/local/lib/python3.8/dist-packages/centrosome/zernike.py", line 101, in construct_zernike_polynomials
result = np.zeros(mask.shape + (nzernikes,), np.complex)
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 11.7 GiB for an array with shape (5120, 5120, 30) and data type complex128
Is your feature request related to a problem? Please describe.
Centrosome's Zernike calculation ends up allocating several complex arrays of image size x 30 (the number of polynomials we calculate), which means for some not-too-huge images (5120^2), we can end up running out of memory to run Zernike calculation even on a 64GB-memory machine. (See example stack trace below).
I'd need to do a deep dive on that code to see if we can in any way sparsify the array, if we NEED the complex data type, if we can maybe calculate each of the 30 polynomials independently, and/or calculate each object in only its neighborhood independently; I imagine at least one if not many of these might be doable. My guess as to why it wasn't already is that it would probably cause a net slowdown on smaller images, but for large image support it might be worth it (and/or we could consider a code path to split based on image size).