lair.parallel.parallelize#
- lair.parallel.parallelize(func, num_processes=1)[source]#
Parallelize a function across an iterable.
- Parameters:
func (function) – The function to parallelize.
num_processes (int or 'max', optional) – The number of processes to use. Uses the minimum of the number of items in the iterable and the number of CPUs requested, and never more than the CPUs this process may run on (a Slurm job’s CPUs, not every CPU of the node; macOS and Windows count the machine’s CPUs). If ‘max’, uses all of those CPUs. Default is 1.
- Returns:
parallelized (function) – A function that will execute the input function in parallel across an iterable.
- Return type:
Notes
Use the call form only, binding the result to a new name:
run = parallelize(func, num_processes=4) results = run(items)
Don’t use it as a decorator (
@parallelize) or rebind the original name (func = parallelize(func, 4)): the worker processes findfuncby its module-level name, so it then no longer pickles.funcmust be defined at module level (no lambdas or nested functions) for the same reason.See also
lair.noaa.CarbonTracker.molefractionsOpens its files in worker processes of its own (one at a time inside a worker of
parallelize); see the CarbonTracker example for the__main__guard a script needs.
Examples
functakes one item and is given each item of the iterable in turn; other arguments are passed to the returned function by keyword:>>> from lair.parallel import parallelize >>> def add(x, b=0): ... return x + b >>> run = parallelize(add, num_processes=1) >>> run([1, 2, 3], b=10) [11, 12, 13]
With
num_processes=4, a script putsfuncat module level and the call underif __name__ == "__main__":, so that each worker can import the script without running the call again.