wim/parfor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

- README update

Browse Source
This commit is contained in:
Wim Pomp
2023-08-08 17:50:20 +02:00
parent 0635c62d42
commit dfee4bc0ea
2 changed files with 25 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

43
README.md
View File

@@ -4,7 +4,7 @@ Take any normal serial but parallelizable for-loop and execute it in parallel us
Don't worry about the technical details of using the multiprocessing module, race conditions, queues, Don't worry about the technical details of using the multiprocessing module, race conditions, queues,
parfor handles all that. parfor handles all that.
Tested on linux on python 2.7 and 3.8 and on Windows and OSX on python 3.8. Tested on linux on python 3.8 and 3.10 and on Windows and OSX on python 3.8.
## Why is parfor better than just using multiprocessing? ## Why is parfor better than just using multiprocessing?
- Easy to use - Easy to use
@@ -12,19 +12,20 @@ Tested on linux on python 2.7 and 3.8 and on Windows and OSX on python 3.8.
- Progress bars are built-in - Progress bars are built-in
## Installation ## Installation
pip install parfor `pip install parfor`
## Usage ## Usage
Parfor decorates a functions and returns the result of that function evaluated in parallel for each iteration of Parfor decorates a functions and returns the result of that function evaluated in parallel for each iteration of
an iterator. an iterator.
## Requires ## Requires
tqdm, dill tqdm, dill, psutil
## Limitations ## Limitations
Objects passed to the pool need to be dillable (dill needs to serialize them). Generators and SwigPyObjects are examples Objects passed to the pool need to be dillable (dill needs to serialize them). Generators and SwigPyObjects are examples
of objects that cannot be used. They can be used however, for the iterator argument when using parfor, but its of objects that cannot be used. They can be used however, for the iterator argument when using parfor, but its
iterations need to be dillable. You might be able to make objects dillable anyhow using dill.register. iterations need to be dillable. You might be able to make objects dillable anyhow using `dill.register` or with
`__reduce__`, `__getstate__`, etc.
The function evaluated in parallel needs to terminate. If parfor hangs after seeming to complete the task, it probably The function evaluated in parallel needs to terminate. If parfor hangs after seeming to complete the task, it probably
is because the individual processes cannot terminate. Importing javabridge (used in python-bioformats) and starting the is because the individual processes cannot terminate. Importing javabridge (used in python-bioformats) and starting the
@@ -36,24 +37,26 @@ On OSX the buffer bar does not work due to limitations of the OS.
## Arguments ## Arguments
### Required: ### Required:
fun: function taking arguments: iteration from iterable, other arguments defined in args & kwargs fun: function taking arguments: iteration from iterable, other arguments defined in args & kwargs
iterable: iterable from which an item is given to fun as a first argument iterable: iterable or iterator from which an item is given to fun as a first argument
### Optional: ### Optional:
args: tuple with other unnamed arguments to fun args: tuple with other unnamed arguments to fun
kwargs: dict with other named arguments to fun kwargs: dict with other named arguments to fun
length: give the length of the iterator in cases where len(iterator) results in an error total: give the length of the iterator in cases where len(iterator) results in an error
desc: string with description of the progress bar desc: string with description of the progress bar
bar: bool enable progress bar bar: bool enable progress bar,
pbar: bool enable buffer indicator bar or a callback function taking the number of passed iterations as an argument
pbar: bool enable buffer indicator bar, or a callback function taking the queue size as an argument
terminator: function which is executed in each worker after all the work is done
rP: ratio workers to cpu cores, default: 1 rP: ratio workers to cpu cores, default: 1
nP: number of workers, default, None, overrides rP if not None nP: number of workers, default, None, overrides rP if not None
number of workers will always be at least 2 serial: execute in series instead of parallel if True, None (default): let pmap decide
serial: switch to serial if number of tasks less than serial, default: 4 qsize: maximum size of the task queue
debug: if an error occurs in an iteration, return the erorr instead of retrying in the main process length: deprecated alias for total
**bar_kwargs: keywords arguments for tqdm.tqdm
### Return ### Return
list with results from applying the decorated function to each iteration of the iterator list with results from applying the function 'fun' to each iteration of the iterable / iterator
specified as the first argument to the function
## Examples ## Examples
### Normal serial for loop ### Normal serial for loop
@@ -139,28 +142,28 @@ pmap maps an iterator to a function like map does, but in parallel
### Using generators ### Using generators
If iterators like lists and tuples are too big for the memory, use generators instead. If iterators like lists and tuples are too big for the memory, use generators instead.
Since generators don't have a predefined length, give parfor the length as an argument (optional). Since generators don't have a predefined length, give parfor the length (total) as an argument (optional).
<< <<
import numpy as np import numpy as np
c = (im for im in imagereader) c = (im for im in imagereader)
@parfor(c, length=len(imagereader)) @parfor(c, total=len(imagereader))
def fun(im): def fun(im):
return np.mean(im) return np.mean(im)
>> [list with means of the images] >> [list with means of the images]
# Extra's # Extra's
## Pmap ## `pmap`
The function parfor decorates, use it like map. The function parfor decorates, use it like `map`.
## Chunks ## `Chunks`
Split a long iterator in bite-sized chunks to parallelize Split a long iterator in bite-sized chunks to parallelize
## Parpool ## `Parpool`
More low-level accessibility to parallel execution. Submit tasks and request the result at any time, More low-level accessibility to parallel execution. Submit tasks and request the result at any time,
(although necessarily submit first, then request a specific task), use different functions and function (although necessarily submit first, then request a specific task), use different functions and function
arguments for different tasks. arguments for different tasks.
## TqdmMeter ## `TqdmMeter`
Meter bar, inherited from tqdm, used for displaying buffers. Meter bar, inherited from tqdm, used for displaying buffers.

4
pyproject.toml
View File

@@ -1,12 +1,12 @@
[tool.poetry] [tool.poetry]
name = "parfor" name = "parfor"
version = "2023.8.0" version = "2023.8.1"
description = "A package to mimic the use of parfor as done in Matlab." description = "A package to mimic the use of parfor as done in Matlab."
authors = ["Wim Pomp <wimpomp@gmail.com>"] authors = ["Wim Pomp <wimpomp@gmail.com>"]
license = "GPLv3" license = "GPLv3"
readme = "README.md" readme = "README.md"
keywords = ["parfor", "concurrency", "multiprocessing", "parallel"] keywords = ["parfor", "concurrency", "multiprocessing", "parallel"]
repository = "https://gitlab.rhpc.nki.nl/LenstraLab/LiveCellAnalysis" repository = "https://github.com/wimpomp/parfor"
[tool.poetry.dependencies] [tool.poetry.dependencies]
python = "^3.5" python = "^3.5"