Skip to content

Conversation

@ninsbl
Copy link
Member

@ninsbl ninsbl commented Jun 18, 2024

Include also #1091

@ninsbl ninsbl added enhancement New feature or request Python Related code is in Python labels Jun 18, 2024
@ninsbl ninsbl requested a review from ecodiv June 18, 2024 21:29
@ecodiv
Copy link
Contributor

ecodiv commented Jun 22, 2024

@ninsbl I tested it using the example from the manual page, resulting in the following error:

gs.run_command(
...     "r.connectivity.distance",
...     flags="k",
...     input="patches_1ha",
...     pop_proxy="area_ha",
...     costs="costs",
...     prefix="hws_connectivity2",
...     cutoff=4500,
...     border_dist=18,
...     conefor_dir="./conefor",
...     nprocs=10,
... )
Traceback (most recent call last):
  File "/home/paulo/.grass8/addons/scripts/r.connectivity.distance", line 1225, in <module>
    sys.exit(main())
  File "/home/paulo/.grass8/addons/scripts/r.connectivity.distance", line 1142, in main
    pool.starmap(compute_distances_parallel, cat_chunks)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 375, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 540, in _handle_tasks
    put(task)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
  File "/usr/local/grassmaster/grass85/etc/python/grass/script/utils.py", line 154, in __getattr__
    return self[key]
KeyError: '__getstate__'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/grassmaster/grass85/etc/python/grass/script/core.py", line 487, in run_command
    return handle_errors(returncode, result=None, args=args, kwargs=kwargs)
  File "/usr/local/grassmaster/grass85/etc/python/grass/script/core.py", line 366, in handle_errors
    raise CalledModuleError(module=module, code=code, returncode=returncode)
  File "/usr/local/grassmaster/grass85/etc/python/grass/exceptions/__init__.py", line 87, in __init__
    err = _("See errors above the traceback or in the error output.")
TypeError: 'int' object is not callable

@ninsbl
Copy link
Member Author

ninsbl commented Jun 22, 2024

Hm... The eror you see is likely related to the number of processes (nprocs). Could you try with nprocs=4 for example?
With 10 cores I get a different error message (with GRASS 8.3) . But with 4 cores it runs fine...

@ecodiv
Copy link
Contributor

ecodiv commented Jun 23, 2024

Hm... The eror you see is likely related to the number of processes (nprocs). Could you try with nprocs=4 for example? With 10 cores I get a different error message (with GRASS 8.3) . But with 4 cores it runs fine...

Unfortunately, for me it doesn't work with 2 or 4 cores either. One core does work.

>>> gs.run_command(
...     "r.connectivity.distance",
...     flags="pk",
...     input="patches_1ha",
...     pop_proxy="area_ha",
...     costs="costs",
...     prefix="hws_connectivity4",
...     cutoff=1000,
...     border_dist=18,
...     nprocs=2
... )
Traceback (most recent call last):
  File "/home/paulo/.grass8/addons/scripts/r.connectivity.distance", line 1225, in <module>
    sys.exit(main())
  File "/home/paulo/.grass8/addons/scripts/r.connectivity.distance", line 1142, in main
    pool.starmap(compute_distances_parallel, cat_chunks)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 375, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 540, in _handle_tasks
    put(task)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
  File "/usr/local/grassmaster/grass85/etc/python/grass/script/utils.py", line 154, in __getattr__
    return self[key]
KeyError: '__getstate__'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/grassmaster/grass85/etc/python/grass/script/core.py", line 487, in run_command
    return handle_errors(returncode, result=None, args=args, kwargs=kwargs)
  File "/usr/local/grassmaster/grass85/etc/python/grass/script/core.py", line 366, in handle_errors
    raise CalledModuleError(module=module, code=code, returncode=returncode)
grass.exceptions.CalledModuleError: Module run `r.connectivity.distance -pk input=patches_1ha pop_proxy=area_ha costs=costs prefix=hws_connectivity4 cutoff=1000 border_dist=18 nprocs=2` ended with an error.
The subprocess ended with a non-zero return code: 1. See errors above the traceback or in the error output.

@echoix
Copy link
Member

echoix commented Jun 23, 2024

Is one of you using a platform other than Linux by any chance? If so, maybe the use of multiprocessing.Pool is using the start method other than fork, like spawn, that requires that what is ran to be Pickle-able. Sometimes the file needs to be reorganized, or even the part that is ran by multiple processes be specially designed and extracted in another file to be executed correctly.

The traceback shown doesn't properly mention the real error for me.

Alternatively, is there any chance that the Python version is 3.12 running on Linux, and using a multiprocessing function using the fork startup method also calls a sub process, such as it raises a DepreciationWarning (it can cause deadlocks and the default startup method will change to spawn in 3.14), and that this depreciationwarning is saved to a results file, making it invalid?

@echoix
Copy link
Member

echoix commented Jun 23, 2024

Using multiprocessing correctly, especially in a cross platform manner, is a little bit trickier than one can imagine

@ecodiv
Copy link
Contributor

ecodiv commented Jun 23, 2024

Alternatively, is there any chance that the Python version is 3.12 running on Linux,

I am running version 3.10.12, the default on Ubuntu 22.04

@ecodiv
Copy link
Contributor

ecodiv commented Jun 23, 2024

@echoix @ninsbl I don't know if it is helpful, but the attached log.txt shows the full debug output when setting debug to level 5.

@echoix echoix changed the title r.connectivty.distance: implement walking distance and parallel processing r.connectivity.distance: implement walking distance and parallel processing Jun 23, 2024
@ecodiv
Copy link
Contributor

ecodiv commented Jun 10, 2025

Hi @ninsbl @echoix I am not sure how to propose changes to code in a pull request, so doing it this way. To avoid a deprecation warning from NumPy, I suggest to replace the code from line 634 - 639

if from_vpatch["vid"].size == 1:
    from_centroid = Centroid(
        v_id=int(from_vpatch["vid"]), c_mapinfo=vpatches.c_mapinfo
    )
    from_x = from_centroid.x
    from_y = from_centroid.y

with

if from_vpatch["vid"].size == 1:
    vid_scalar = from_vpatch["vid"].item()
    from_centroid = Centroid(v_id=int(vid_scalar), c_mapinfo=vpatches.c_mapinfo)
    from_x = from_centroid.x
    from_y = from_centroid.y

@ecodiv
Copy link
Contributor

ecodiv commented Jun 10, 2025

Running with nprocs=1

When running the example from the manual page with nproc=1, I am getting the warning:
WARNING: Values in column <cat> will be overwritten

I guess it is when writing values to output vector maps, perhaps in line 590 and 597? This warning pops up irrespective of the number of nprocs.

Running with nprocs=4

When running with nprocs=4, I am getting an output, but with the following warning

WARNING: WARNING: YouYou  dodo  notnot  havehave  accessaccess  toto  mapsetmapset  tmp_bLZm4hKCH5on_2.tmp_bLZm4hKCH5on_1.  RunRun  g.mapsetsg.mapsets
         
         (under(under  settingssettings  menu)menu)  toto  changechange  mapsetmapset  accessaccess

WARNING: You do not have access to mapset tmp_bLZm4hKCH5on_3. Run g.mapsets
         (under settings menu) to change mapset access
WARNING: You do not have access to mapset tmp_bLZm4hKCH5on_4. Run g.mapsets
         (under settings menu) to change mapset access

And the temporary layers tmp_bLZm4hKCH5on_patches_boundary, tmp_bLZm4hKCH5on_patches_pol and tmp_SWle0jlDUOKO_patches_boundary are not removed. In addition, no cost distance raster layers are created.

Running with nprocs=2

I get the following error when running with nprocs=2

Building topology for vector map <hws_4_edges@tmp_jhpwdYPSDPjM_1>...
Registering primitives...
Building topology for vector map <hws_4_vertices@tmp_jhpwdYPSDPjM_1>...
Registering primitives...
Building topology for vector map <hws_4_edges@tmp_jhpwdYPSDPjM_2>...
Registering primitives...
Building topology for vector map <hws_4_vertices@tmp_jhpwdYPSDPjM_2>...
Registering primitives...
WARNING: WARNING: <hws_4_patch_105_cost_dist><hws_4_patch_100_cost_dist>  alreadyalready  existsexists  andand  overwriteoverwrite  isis  notnot
         
         enabled.enabled.  Skipping.Skipping.

ERROR: ERROR: AllAll  inputsinputs  hadhad  toto  bebe  skipped.skipped.  NothingNothing  toto  copy.copy.  SeeSee  thethe  warningwarning
       
       messagesmessages  aboveabove  forfor  details.details.

Traceback (most recent call last):
  File "/home/paulo/.grass8/addons/scripts/g.copyall", line 130, in <module>
    main()
  File "/home/paulo/.grass8/addons/scripts/g.copyall", line 122, in main
    gs.run_command("g.copy", overwrite=overwrite, **params)
  File "/usr/local/grassmaster/grass85/etc/python/grass/script/core.py", line 522, in run_command
    return handle_errors(returncode, result=None, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/grassmaster/grass85/etc/python/grass/script/core.py", line 375, in handle_errors
    raise CalledModuleError(module=module, code=code, returncode=returncode)
grass.exceptions.CalledModuleError: Module run `g.copy rast=hws_4_patch_100_cost_dist@tmp_jhpwdYPSDPjM_2,hws_4_patch_100_cost_dist` ended with an error.
The subprocess ended with a non-zero return code: 1. See errors above the traceback or in the error output.
Traceback (most recent call last):
  File "/home/paulo/.grass8/addons/scripts/g.copyall", line 130, in <module>
    main()
  File "/home/paulo/.grass8/addons/scripts/g.copyall", line 122, in main
    gs.run_command("g.copy", overwrite=overwrite, **params)
  File "/usr/local/grassmaster/grass85/etc/python/grass/script/core.py", line 522, in run_command
    return handle_errors(returncode, result=None, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/grassmaster/grass85/etc/python/grass/script/core.py", line 375, in handle_errors
    raise CalledModuleError(module=module, code=code, returncode=returncode)
grass.exceptions.CalledModuleError: Module run `g.copy rast=hws_4_patch_105_cost_dist@tmp_jhpwdYPSDPjM_1,hws_4_patch_105_cost_dist` ended with an error.
The subprocess ended with a non-zero return code: 1. See errors above the traceback or in the error output.
Exception in thread Thread-3 (_handle_results):
Traceback (most recent call last):
  File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.12/multiprocessing/pool.py", line 579, in _handle_results
    task = get()
           ^^^^^
  File "/usr/lib/python3.12/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: CalledModuleError.__init__() missing 3 required positional arguments: 'module', 'code', and 'returncode'

With nprocs=8

Registering primitives...
WARNING: You do not have access to mapset tmp_JwwsudshJ8du_2. Run g.mapsets
         (under settings menu) to change mapset access
WARNING: You do not have access to mapset tmp_JwwsudshJ8du_1. Run g.mapsets
         (under settings menu) to change mapset access
WARNING: You do not have access to mapset tmp_JwwsudshJ8du_8. Run g.mapsets
         (under settings menu) to change mapset access
WARNING: You do not have access to mapset tmp_JwwsudshJ8du_5. Run g.mapsets
         (under settings menu) to change mapset access
WARNING: You do not have access to mapset tmp_JwwsudshJ8du_4. Run g.mapsets
         (under settings menu) to change mapset access
WARNING: WARNING: YouYou  dodo  notnot  havehave  accessaccess  toto  mapsetmapset  tmp_JwwsudshJ8du_6.tmp_JwwsudshJ8du_7.  RunRun  g.mapsetsg.mapsets
         
         (under(under  settingssettings  menu)menu)  toto  changechange  mapsetmapset  accessaccess

WARNING: <hws_4_patch_105_cost_dist> already exists and overwrite is not
         enabled. Skipping.
ERROR: All inputs had to be skipped. Nothing to copy. See the warning
       messages above for details.
Traceback (most recent call last):
  File "/home/paulo/.grass8/addons/scripts/g.copyall", line 130, in <module>
    main()
  File "/home/paulo/.grass8/addons/scripts/g.copyall", line 122, in main
    gs.run_command("g.copy", overwrite=overwrite, **params)
  File "/usr/local/grassmaster/grass85/etc/python/grass/script/core.py", line 522, in run_command
    return handle_errors(returncode, result=None, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/grassmaster/grass85/etc/python/grass/script/core.py", line 375, in handle_errors
    raise CalledModuleError(module=module, code=code, returncode=returncode)
grass.exceptions.CalledModuleError: Module run `g.copy rast=hws_4_patch_105_cost_dist@tmp_JwwsudshJ8du_3,hws_4_patch_105_cost_dist` ended with an error.
The subprocess ended with a non-zero return code: 1. See errors above the traceback or in the error output.
Exception in thread Thread-3 (_handle_results):
Traceback (most recent call last):
  File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.12/threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.12/multiprocessing/pool.py", line 579, in _handle_results
    task = get()
           ^^^^^
  File "/usr/lib/python3.12/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: CalledModuleError.__init__() missing 3 required positional arguments: 'module', 'code', and 'returncode'
^CProcess ForkPoolWorker-4:
Process ForkPoolWorker-1:
Process ForkPoolWorker-7:
Process ForkPoolWorker-8:
Process ForkPoolWorker-5:
Process ForkPoolWorker-6:
Process ForkPoolWorker-2:
Process ForkPoolWorker-3:
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.12/multiprocessing/pool.py", line 114, in worker
    task = get()
           ^^^^^
  File "/usr/lib/python3.12/multiprocessing/queues.py", line 386, in get
    with self._rlock:
Traceback (most recent call last):
  File "/usr/lib/python3.12/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
Traceback (most recent call last):
Traceback (most recent call last):
KeyboardInterrupt
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.12/multiprocessing/pool.py", line 114, in worker
    task = get()
           ^^^^^
  File "/usr/lib/python3.12/multiprocessing/queues.py", line 386, in get
    with self._rlock:
  File "/usr/lib/python3.12/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.12/multiprocessing/pool.py", line 114, in worker
    task = get()
           ^^^^^
  File "/usr/lib/python3.12/multiprocessing/queues.py", line 386, in get
    with self._rlock:
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.12/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
Traceback (most recent call last):
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
KeyboardInterrupt
  File "/usr/lib/python3.12/multiprocessing/pool.py", line 114, in worker
    task = get()
           ^^^^^
  File "/usr/lib/python3.12/multiprocessing/queues.py", line 386, in get
    with self._rlock:
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.12/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/pool.py", line 114, in worker
    task = get()
           ^^^^^
  File "/usr/lib/python3.12/multiprocessing/queues.py", line 386, in get
    with self._rlock:
  File "/usr/lib/python3.12/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
KeyboardInterrupt
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.12/multiprocessing/pool.py", line 114, in worker
    task = get()
           ^^^^^
  File "/usr/lib/python3.12/multiprocessing/queues.py", line 386, in get
    with self._rlock:
  File "/usr/lib/python3.12/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.12/multiprocessing/pool.py", line 114, in worker
    task = get()
           ^^^^^
  File "/usr/lib/python3.12/multiprocessing/queues.py", line 387, in get
    res = self._reader.recv_bytes()
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/connection.py", line 430, in _recv_bytes
    buf = self._recv(4)
          ^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/connection.py", line 395, in _recv
    chunk = read(handle, remaining)
            ^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.12/multiprocessing/pool.py", line 114, in worker
    task = get()
           ^^^^^
  File "/usr/lib/python3.12/multiprocessing/queues.py", line 386, in get
    with self._rlock:
  File "/usr/lib/python3.12/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
Traceback (most recent call last):
  File "/home/paulo/.grass8/addons/scripts/r.connectivity.distance", line 1144, in main
    pool.map(
  File "/usr/lib/python3.12/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/pool.py", line 768, in get
    self.wait(timeout)
  File "/usr/lib/python3.12/multiprocessing/pool.py", line 765, in wait
    self._event.wait(timeout)
  File "/usr/lib/python3.12/threading.py", line 655, in wait
    signaled = self._cond.wait(timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/threading.py", line 355, in wait
    waiter.acquire()
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/paulo/.grass8/addons/scripts/r.connectivity.distance", line 1225, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/paulo/.grass8/addons/scripts/r.connectivity.distance", line 1141, in main
    with Pool(nprocs) as pool:
  File "/usr/lib/python3.12/multiprocessing/pool.py", line 739, in __exit__
    self.terminate()
  File "/usr/lib/python3.12/multiprocessing/pool.py", line 657, in terminate
    self._terminate()
  File "/usr/lib/python3.12/multiprocessing/util.py", line 227, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/pool.py", line 698, in _terminate_pool
    raise AssertionError(
AssertionError: Cannot have cache with result_handler not alive

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Python Related code is in Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants