wim/parfor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
62 Commits 1 Branch 0 Tags
5528b54edeb721b3619314c92252d895f38a92dc
Commit Graph

62 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Wim Pomp
5528b54ede Now based on ray, which enables nested parallel computations. 2026-01-08 00:48:14 +01:00
Wim Pomp
a84bf2d29e Merge remote-tracking branch 'origin/master' 2025-01-06 18:14:40 +01:00
Wim Pomp
d0bc749bb4 - add SharedArray to share numpy arrays 
- fix async call_later when no event loop is available
 2025-01-06 18:14:18 +01:00
Wim Pomp
326a06f9da - add SharedArray to share numpy arrays 
- fix async call_later when no event loop is available
 2025-01-06 18:01:55 +01:00
Wim Pomp
eb92ce006d - make cpu_count a field to PoolSingleton for easy global configuration of number of processes 
- remove TypeVars
- manually wrap parfor and pmap
- also redirect output when retrieving task
 2024-12-20 16:43:02 +01:00
Wim Pomp
31e07b49eb - statically determine gil/nogil, otherwise we cannot subclass some things 2024-12-05 12:57:46 +01:00
Wim Pomp
7291468fb7 - add async callback to close pool after 10 minutes of inactivity 2024-11-05 15:05:10 +01:00
Wim Pomp
46bd419d3d - add Task class for typing again 2024-11-04 14:52:46 +01:00
Wim Pomp
c47650fc30 - add close method to PoolSingleton class again 2024-10-22 11:23:16 +02:00
Wim Pomp
396a1a791d - gmap bugfix 2024-10-21 10:25:17 +02:00
Wim Pomp
d6afbc3caf - add PoolSingleton to __init__ 
- update readme
 2024-10-20 13:10:49 +02:00
Wim Pomp
57b3fbe2b2 - nogil version (selected automatically) which uses threads instead of processes 2024-10-20 12:45:51 +02:00
Wim Pomp
b3aabc68ab - nogil version (selected automatically) which uses threads instead of processes 2024-10-20 12:45:08 +02:00
Wim Pomp
f43734c8be - fix zeros length iterator bug 
- fix bug where sometimes not all results are returned
 2024-09-13 14:53:42 +02:00
Wim Pomp
fb7757828f - fix serial computation 
- some more tests
 2024-09-11 13:08:03 +02:00
Wim Pomp
4d80316244 - add gmap: function like pmap, but returning a generator instead of a list 
- add arguments for returning results out/in order and returning result indices
 2024-09-05 18:37:47 +02:00
Wim Pomp
29203dd128 - redirect stdout and stderr in child processes to devnull instead of None 
- add parfor.__version__
 2024-07-22 17:32:17 +02:00
Wim Pomp
9ce96a7e61 - redirect stdout and stderr in child processes 2024-07-08 17:57:23 +02:00
Wim Pomp
b609df8b52 - bugfix: do not restart pool when n_processes = None 2024-05-28 16:35:14 +02:00
Wim Pomp
9783c1d1f2 - introduce n_processes to change the number of processes in the pool 2024-05-24 16:57:35 +02:00
Wim Pomp
ac4d599646 - minimum python: 3.10 
- typing
- task_error bug fix
- remove some deprecated functions
 2024-04-26 18:32:12 +02:00
Wim Pomp
42746d21eb - make ParPoolSingleton.close a class method. 2024-03-19 18:52:20 +01:00
Wim Pomp
6fefdb9be0 - bugfix for argument bar=False 2024-02-09 13:32:18 +01:00
Wim Pomp
bb3b87b913 - catch and ignore errors when emptying queue 
- timeout on closing pool
- close manager on closing pool
 2024-01-29 11:25:22 +01:00
Wim Pomp
0b0c7a798f - Shutdown workers after 10 minutes of inactivity. Restart pool when needed again. 2023-11-18 13:38:01 +01:00
Wim Pomp
ad8d9a4efb - Protect reference counter with a lock because changes to a multiprocessing manager dict are not atomic. 2023-11-08 12:50:30 +01:00
Wim Pomp
098d1810c3 - Bugfix 2023-11-07 16:51:02 +01:00
Wim Pomp
f22b682b2a - skip KeyboardInterrupts in workers 
- silence some warnings
 2023-11-07 15:04:54 +01:00
Wim Pomp
e936292905 - Implement reference counting to make sure the parent process does not trash the item before all children trashed it. 2023-10-27 17:48:52 +02:00
Wim Pomp
265470e0ac - Use a shared memory approach. 
- Track unpickled objects by CouldNotBePickled class.
- Bump minimal python to 3.8.
 2023-10-12 14:46:42 +02:00
Wim Pomp
0263b6a4c7 - Copy previous task to prevent unnecessary pickling. 2023-09-13 17:04:55 +02:00
Wim Pomp
f3302f6dba - Use a singleton pool to prevent lengthy restarts of the pool, this also means that arguments for pool size have gone. 
- Removed qbar and TqdmMeter.
- Wrap chunked function for better error messages.
 2023-09-08 17:19:35 +02:00
Wim Pomp
92f162b7d5 - Make processes non-daemonic 
- Deprecate terminator argument
 2023-08-16 16:46:50 +02:00
Wim Pomp
dfee4bc0ea - README update 2023-08-08 17:50:20 +02:00
Wim Pomp
0635c62d42 - bugfix: some iterables caused an error 
- pytest tests
- deprecation warnings
- use poetry for installs
 2023-08-08 15:05:31 +02:00
Wim Pomp
7e0c0cc45d - bugfix in parpool.get_newest 
- make TqdmMeter bar decrease when necessary
 2022-06-28 11:08:37 +02:00
Wim Pomp
84f4afbba2 Merge remote-tracking branch 'origin/master' 2022-05-03 15:56:20 +02:00
Wim Pomp
32c5560306 - rename some things to conform to PEP 
- pmap and parfor now automatically divide the task in chunks: less pickling is needed this way
- pickle code in a new file: prevent some unpickleble things from getting pickled
- Chunks now supports generators when the length keyword argument is supplied
- TqdmMeter can have a description
 2022-05-03 15:56:13 +02:00
Wim Pomp
d2c226a37e - rename some things to conform to PEP 
- pmap and parfor now automatically divide the task in chunks: less pickling is needed this way
- pickle code in a new file: prevent some unpickleble things from getting pickled
- Chunks now supports generators when the length keyword argument is supplied
- TqdmMeter can have a description
 2022-05-03 15:55:11 +02:00
Wim Pomp
865ec70d97 - bug fix in killed process detection 2022-03-22 15:36:07 +01:00
Wim Pomp
fcf451cb23 - drop support for python 2 
- automatically kill javabridge (if it exists) at the end of a process
- better hashing and tasking
 2022-03-03 12:43:00 +01:00
Wim Pomp
4bddca82eb - option for a callback function to update a user defined bar 2021-07-29 21:04:28 +02:00
Wim Pomp
8ed1a5500e - chunks: kwargs to set number of chunks relative to number of cpus 2021-07-17 10:54:00 +02:00
Wim Pomp
bc2ae2ea25 - Make progress bar update with chunksize when using chunks. 
- Catch exceptions, but not keyboard interrupts.
 2021-05-29 11:09:54 +02:00
Wim Pomp
9caeb06c2b - Bugfix 2021-05-25 10:51:14 +02:00
Wim Pomp
6c9c18746b - Make chunks a class, so we can do len(chunks(...)) 2021-05-24 18:59:32 +02:00
Wim Pomp
95ae325028 - Better chunks function, option for both number of chunks or size of chunks. 2021-05-24 18:19:14 +02:00
Wim Pomp
5d57081713 - Better and faster error handling. 2021-03-19 20:39:38 +01:00
Wim Pomp
ddc0189bad - Add get_newest method to parpool. 2021-03-06 00:09:16 +01:00
Wim Pomp
6cce9e9b8e - Better error handling 2021-03-05 22:27:00 +01:00