- Use a singleton pool to prevent lengthy restarts of the pool, this also means that arguments for pool size have gone.

- Removed qbar and TqdmMeter.
- Wrap chunked function for better error messages.
This commit is contained in:
Wim Pomp
2023-09-08 17:19:35 +02:00
parent 92f162b7d5
commit f3302f6dba
6 changed files with 246 additions and 246 deletions

View File

@@ -1,10 +1,12 @@
[![pytest](https://github.com/wimpomp/parfor/actions/workflows/pytest.yml/badge.svg)](https://github.com/wimpomp/parfor/actions/workflows/pytest.yml)
# Parfor
Used to parallelize for-loops using parfor in Matlab? This package allows you to do the same in python.
Take any normal serial but parallelizable for-loop and execute it in parallel using easy syntax.
Don't worry about the technical details of using the multiprocessing module, race conditions, queues,
parfor handles all that.
Tested on linux on python 3.8 and 3.10 and on Windows and OSX on python 3.8.
Tested on linux, Windows and OSX with python 3.10.
## Why is parfor better than just using multiprocessing?
- Easy to use
@@ -27,8 +29,6 @@ of objects that cannot be used. They can be used however, for the iterator argum
iterations need to be dillable. You might be able to make objects dillable anyhow using `dill.register` or with
`__reduce__`, `__getstate__`, etc.
On OSX the buffer bar does not work due to limitations of the OS.
## Arguments
### Required:
fun: function taking arguments: iteration from iterable, other arguments defined in args & kwargs
@@ -154,10 +154,7 @@ The function parfor decorates, use it like `map`.
## `Chunks`
Split a long iterator in bite-sized chunks to parallelize
## `Parpool`
## `ParPool`
More low-level accessibility to parallel execution. Submit tasks and request the result at any time,
(although necessarily submit first, then request a specific task), use different functions and function
arguments for different tasks.
## `TqdmMeter`
Meter bar, inherited from tqdm, used for displaying buffers.