- README update

This commit is contained in:
Wim Pomp
2023-08-08 17:50:20 +02:00
parent 0635c62d42
commit dfee4bc0ea
2 changed files with 25 additions and 22 deletions

View File

@@ -4,7 +4,7 @@ Take any normal serial but parallelizable for-loop and execute it in parallel us
Don't worry about the technical details of using the multiprocessing module, race conditions, queues,
parfor handles all that.
Tested on linux on python 2.7 and 3.8 and on Windows and OSX on python 3.8.
Tested on linux on python 3.8 and 3.10 and on Windows and OSX on python 3.8.
## Why is parfor better than just using multiprocessing?
- Easy to use
@@ -12,19 +12,20 @@ Tested on linux on python 2.7 and 3.8 and on Windows and OSX on python 3.8.
- Progress bars are built-in
## Installation
pip install parfor
`pip install parfor`
## Usage
Parfor decorates a functions and returns the result of that function evaluated in parallel for each iteration of
an iterator.
## Requires
tqdm, dill
tqdm, dill, psutil
## Limitations
Objects passed to the pool need to be dillable (dill needs to serialize them). Generators and SwigPyObjects are examples
of objects that cannot be used. They can be used however, for the iterator argument when using parfor, but its
iterations need to be dillable. You might be able to make objects dillable anyhow using dill.register.
iterations need to be dillable. You might be able to make objects dillable anyhow using `dill.register` or with
`__reduce__`, `__getstate__`, etc.
The function evaluated in parallel needs to terminate. If parfor hangs after seeming to complete the task, it probably
is because the individual processes cannot terminate. Importing javabridge (used in python-bioformats) and starting the
@@ -36,24 +37,26 @@ On OSX the buffer bar does not work due to limitations of the OS.
## Arguments
### Required:
fun: function taking arguments: iteration from iterable, other arguments defined in args & kwargs
iterable: iterable from which an item is given to fun as a first argument
iterable: iterable or iterator from which an item is given to fun as a first argument
### Optional:
args: tuple with other unnamed arguments to fun
kwargs: dict with other named arguments to fun
length: give the length of the iterator in cases where len(iterator) results in an error
total: give the length of the iterator in cases where len(iterator) results in an error
desc: string with description of the progress bar
bar: bool enable progress bar
pbar: bool enable buffer indicator bar
bar: bool enable progress bar,
or a callback function taking the number of passed iterations as an argument
pbar: bool enable buffer indicator bar, or a callback function taking the queue size as an argument
terminator: function which is executed in each worker after all the work is done
rP: ratio workers to cpu cores, default: 1
nP: number of workers, default, None, overrides rP if not None
number of workers will always be at least 2
serial: switch to serial if number of tasks less than serial, default: 4
debug: if an error occurs in an iteration, return the erorr instead of retrying in the main process
serial: execute in series instead of parallel if True, None (default): let pmap decide
qsize: maximum size of the task queue
length: deprecated alias for total
**bar_kwargs: keywords arguments for tqdm.tqdm
### Return
list with results from applying the decorated function to each iteration of the iterator
specified as the first argument to the function
list with results from applying the function 'fun' to each iteration of the iterable / iterator
## Examples
### Normal serial for loop
@@ -139,28 +142,28 @@ pmap maps an iterator to a function like map does, but in parallel
### Using generators
If iterators like lists and tuples are too big for the memory, use generators instead.
Since generators don't have a predefined length, give parfor the length as an argument (optional).
Since generators don't have a predefined length, give parfor the length (total) as an argument (optional).
<<
import numpy as np
c = (im for im in imagereader)
@parfor(c, length=len(imagereader))
@parfor(c, total=len(imagereader))
def fun(im):
return np.mean(im)
>> [list with means of the images]
# Extra's
## Pmap
The function parfor decorates, use it like map.
## `pmap`
The function parfor decorates, use it like `map`.
## Chunks
## `Chunks`
Split a long iterator in bite-sized chunks to parallelize
## Parpool
## `Parpool`
More low-level accessibility to parallel execution. Submit tasks and request the result at any time,
(although necessarily submit first, then request a specific task), use different functions and function
arguments for different tasks.
## TqdmMeter
## `TqdmMeter`
Meter bar, inherited from tqdm, used for displaying buffers.