lair.parallel.parallelize#

lair.parallel.parallelize(func, num_processes=1)[source]#

Parallelize a function across an iterable.

Parameters:
  • func (function) – The function to parallelize.

  • num_processes (int or 'max', optional) – The number of processes to use. Uses the minimum of the number of items in the iterable and the number of CPUs requested, and never more than the CPUs this process may run on (a Slurm job’s CPUs, not every CPU of the node; macOS and Windows count the machine’s CPUs). If ‘max’, uses all of those CPUs. Default is 1.

Returns:

parallelized (function) – A function that will execute the input function in parallel across an iterable.

Return type:

Callable

Notes

Use the call form only, binding the result to a new name:

run = parallelize(func, num_processes=4)
results = run(items)

Don’t use it as a decorator (@parallelize) or rebind the original name (func = parallelize(func, 4)): the worker processes find func by its module-level name, so it then no longer pickles. func must be defined at module level (no lambdas or nested functions) for the same reason.

See also

lair.noaa.CarbonTracker.molefractions

Opens its files in worker processes of its own (one at a time inside a worker of parallelize); see the CarbonTracker example for the __main__ guard a script needs.

Examples

func takes one item and is given each item of the iterable in turn; other arguments are passed to the returned function by keyword:

>>> from lair.parallel import parallelize
>>> def add(x, b=0):
...     return x + b
>>> run = parallelize(add, num_processes=1)
>>> run([1, 2, 3], b=10)
[11, 12, 13]

With num_processes=4, a script puts func at module level and the call under if __name__ == "__main__":, so that each worker can import the script without running the call again.