wim/parfor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
73 Commits 1 Branch 0 Tags
bb74d6e3b01a6407ba1b67a62c3cdea2aa67848f
Commit Graph

73 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
w.pomp
bb74d6e3b0 - add close method to PoolSingleton for backwards compatibility 2026-01-21 15:27:45 +01:00
Wim Pomp
9fdbf49b5c - n_processes can now be changed each time 
- block adding tasks if pool is busy
 2026-01-09 11:37:55 +01:00
Wim Pomp
1098239af9 recurse when pickling objects with dill 2026-01-08 20:14:08 +01:00
Wim Pomp
81d0362540 - separate output from progress bar 2026-01-08 17:18:26 +01:00
Wim Pomp
2cff1a1515 - add custom pickler back in 2026-01-08 16:25:24 +01:00
Wim Pomp
f3737ffd44 - publish to pypi using github action 2026-01-08 11:15:23 +01:00
Wim Pomp
d47db7aa14 - publish to pypi using github action 2026-01-08 11:07:19 +01:00
Wim Pomp
0ccd09f079 - publish to pypi using github action 2026-01-08 10:57:50 +01:00
Wim Pomp
43a0cf68b5 - add allow_output option 
- update readme
 2026-01-08 10:42:50 +01:00
Wim Pomp
6341561380 update pyproject.toml 2026-01-08 01:03:59 +01:00
Wim Pomp
310616d583 update pyproject.toml 2026-01-08 01:00:53 +01:00
Wim Pomp
5528b54ede Now based on ray, which enables nested parallel computations. 2026-01-08 00:48:14 +01:00
Wim Pomp
a84bf2d29e Merge remote-tracking branch 'origin/master' 2025-01-06 18:14:40 +01:00
Wim Pomp
d0bc749bb4 - add SharedArray to share numpy arrays 
- fix async call_later when no event loop is available
 2025-01-06 18:14:18 +01:00
Wim Pomp
326a06f9da - add SharedArray to share numpy arrays 
- fix async call_later when no event loop is available
 2025-01-06 18:01:55 +01:00
Wim Pomp
eb92ce006d - make cpu_count a field to PoolSingleton for easy global configuration of number of processes 
- remove TypeVars
- manually wrap parfor and pmap
- also redirect output when retrieving task
 2024-12-20 16:43:02 +01:00
Wim Pomp
31e07b49eb - statically determine gil/nogil, otherwise we cannot subclass some things 2024-12-05 12:57:46 +01:00
Wim Pomp
7291468fb7 - add async callback to close pool after 10 minutes of inactivity 2024-11-05 15:05:10 +01:00
Wim Pomp
46bd419d3d - add Task class for typing again 2024-11-04 14:52:46 +01:00
Wim Pomp
c47650fc30 - add close method to PoolSingleton class again 2024-10-22 11:23:16 +02:00
Wim Pomp
396a1a791d - gmap bugfix 2024-10-21 10:25:17 +02:00
Wim Pomp
d6afbc3caf - add PoolSingleton to __init__ 
- update readme
 2024-10-20 13:10:49 +02:00
Wim Pomp
57b3fbe2b2 - nogil version (selected automatically) which uses threads instead of processes 2024-10-20 12:45:51 +02:00
Wim Pomp
b3aabc68ab - nogil version (selected automatically) which uses threads instead of processes 2024-10-20 12:45:08 +02:00
Wim Pomp
f43734c8be - fix zeros length iterator bug 
- fix bug where sometimes not all results are returned
 2024-09-13 14:53:42 +02:00
Wim Pomp
fb7757828f - fix serial computation 
- some more tests
 2024-09-11 13:08:03 +02:00
Wim Pomp
4d80316244 - add gmap: function like pmap, but returning a generator instead of a list 
- add arguments for returning results out/in order and returning result indices
 2024-09-05 18:37:47 +02:00
Wim Pomp
29203dd128 - redirect stdout and stderr in child processes to devnull instead of None 
- add parfor.__version__
 2024-07-22 17:32:17 +02:00
Wim Pomp
9ce96a7e61 - redirect stdout and stderr in child processes 2024-07-08 17:57:23 +02:00
Wim Pomp
b609df8b52 - bugfix: do not restart pool when n_processes = None 2024-05-28 16:35:14 +02:00
Wim Pomp
9783c1d1f2 - introduce n_processes to change the number of processes in the pool 2024-05-24 16:57:35 +02:00
Wim Pomp
ac4d599646 - minimum python: 3.10 
- typing
- task_error bug fix
- remove some deprecated functions
 2024-04-26 18:32:12 +02:00
Wim Pomp
42746d21eb - make ParPoolSingleton.close a class method. 2024-03-19 18:52:20 +01:00
Wim Pomp
6fefdb9be0 - bugfix for argument bar=False 2024-02-09 13:32:18 +01:00
Wim Pomp
bb3b87b913 - catch and ignore errors when emptying queue 
- timeout on closing pool
- close manager on closing pool
 2024-01-29 11:25:22 +01:00
Wim Pomp
0b0c7a798f - Shutdown workers after 10 minutes of inactivity. Restart pool when needed again. 2023-11-18 13:38:01 +01:00
Wim Pomp
ad8d9a4efb - Protect reference counter with a lock because changes to a multiprocessing manager dict are not atomic. 2023-11-08 12:50:30 +01:00
Wim Pomp
098d1810c3 - Bugfix 2023-11-07 16:51:02 +01:00
Wim Pomp
f22b682b2a - skip KeyboardInterrupts in workers 
- silence some warnings
 2023-11-07 15:04:54 +01:00
Wim Pomp
e936292905 - Implement reference counting to make sure the parent process does not trash the item before all children trashed it. 2023-10-27 17:48:52 +02:00
Wim Pomp
265470e0ac - Use a shared memory approach. 
- Track unpickled objects by CouldNotBePickled class.
- Bump minimal python to 3.8.
 2023-10-12 14:46:42 +02:00
Wim Pomp
0263b6a4c7 - Copy previous task to prevent unnecessary pickling. 2023-09-13 17:04:55 +02:00
Wim Pomp
f3302f6dba - Use a singleton pool to prevent lengthy restarts of the pool, this also means that arguments for pool size have gone. 
- Removed qbar and TqdmMeter.
- Wrap chunked function for better error messages.
 2023-09-08 17:19:35 +02:00
Wim Pomp
92f162b7d5 - Make processes non-daemonic 
- Deprecate terminator argument
 2023-08-16 16:46:50 +02:00
Wim Pomp
dfee4bc0ea - README update 2023-08-08 17:50:20 +02:00
Wim Pomp
0635c62d42 - bugfix: some iterables caused an error 
- pytest tests
- deprecation warnings
- use poetry for installs
 2023-08-08 15:05:31 +02:00
Wim Pomp
7e0c0cc45d - bugfix in parpool.get_newest 
- make TqdmMeter bar decrease when necessary
 2022-06-28 11:08:37 +02:00
Wim Pomp
84f4afbba2 Merge remote-tracking branch 'origin/master' 2022-05-03 15:56:20 +02:00
Wim Pomp
32c5560306 - rename some things to conform to PEP 
- pmap and parfor now automatically divide the task in chunks: less pickling is needed this way
- pickle code in a new file: prevent some unpickleble things from getting pickled
- Chunks now supports generators when the length keyword argument is supplied
- TqdmMeter can have a description
 2022-05-03 15:56:13 +02:00
Wim Pomp
d2c226a37e - rename some things to conform to PEP 
- pmap and parfor now automatically divide the task in chunks: less pickling is needed this way
- pickle code in a new file: prevent some unpickleble things from getting pickled
- Chunks now supports generators when the length keyword argument is supplied
- TqdmMeter can have a description
 2022-05-03 15:55:11 +02:00