wim/parfor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Commit Graph

  • 8619c0fb34 - add some caching to prevent repeated ray.put and pickling master w.pomp 2026-01-22 20:11:13 +01:00
  • 879da12628 - disable ray dashboard w.pomp 2026-01-21 16:26:41 +01:00
  • bb74d6e3b0 - add close method to PoolSingleton for backwards compatibility w.pomp 2026-01-21 15:27:45 +01:00
  • 9fdbf49b5c - n_processes can now be changed each time - block adding tasks if pool is busy Wim Pomp 2026-01-09 11:37:55 +01:00
  • 1098239af9 recurse when pickling objects with dill Wim Pomp 2026-01-08 20:14:08 +01:00
  • 81d0362540 - separate output from progress bar Wim Pomp 2026-01-08 17:18:26 +01:00
  • 2cff1a1515 - add custom pickler back in Wim Pomp 2026-01-08 16:25:24 +01:00
  • f3737ffd44 - publish to pypi using github action Wim Pomp 2026-01-08 11:15:23 +01:00
  • d47db7aa14 - publish to pypi using github action Wim Pomp 2026-01-08 11:07:19 +01:00
  • 0ccd09f079 - publish to pypi using github action Wim Pomp 2026-01-08 10:57:50 +01:00
  • 43a0cf68b5 - add allow_output option - update readme Wim Pomp 2026-01-08 10:42:50 +01:00
  • 6341561380 update pyproject.toml Wim Pomp 2026-01-08 01:03:59 +01:00
  • 310616d583 update pyproject.toml Wim Pomp 2026-01-08 01:00:53 +01:00
  • 5528b54ede Now based on ray, which enables nested parallel computations. Wim Pomp 2026-01-08 00:48:14 +01:00
  • a84bf2d29e Merge remote-tracking branch 'origin/master' Wim Pomp 2025-01-06 18:14:40 +01:00
  • d0bc749bb4 - add SharedArray to share numpy arrays - fix async call_later when no event loop is available Wim Pomp 2025-01-06 18:01:55 +01:00
  • 326a06f9da - add SharedArray to share numpy arrays - fix async call_later when no event loop is available Wim Pomp 2025-01-06 18:01:55 +01:00
  • eb92ce006d - make cpu_count a field to PoolSingleton for easy global configuration of number of processes - remove TypeVars - manually wrap parfor and pmap - also redirect output when retrieving task Wim Pomp 2024-12-20 16:43:02 +01:00
  • 31e07b49eb - statically determine gil/nogil, otherwise we cannot subclass some things Wim Pomp 2024-12-05 12:57:46 +01:00
  • 7291468fb7 - add async callback to close pool after 10 minutes of inactivity Wim Pomp 2024-11-05 15:05:10 +01:00
  • 46bd419d3d - add Task class for typing again Wim Pomp 2024-11-04 14:52:46 +01:00
  • c47650fc30 - add close method to PoolSingleton class again Wim Pomp 2024-10-22 11:23:16 +02:00
  • 396a1a791d - gmap bugfix Wim Pomp 2024-10-21 10:25:17 +02:00
  • d6afbc3caf - add PoolSingleton to __init__ - update readme Wim Pomp 2024-10-20 13:10:49 +02:00
  • 57b3fbe2b2 - nogil version (selected automatically) which uses threads instead of processes Wim Pomp 2024-10-20 12:45:51 +02:00
  • b3aabc68ab - nogil version (selected automatically) which uses threads instead of processes Wim Pomp 2024-10-20 12:45:08 +02:00
  • f43734c8be - fix zeros length iterator bug - fix bug where sometimes not all results are returned Wim Pomp 2024-09-13 14:53:42 +02:00
  • fb7757828f - fix serial computation - some more tests Wim Pomp 2024-09-11 13:08:03 +02:00
  • 4d80316244 - add gmap: function like pmap, but returning a generator instead of a list - add arguments for returning results out/in order and returning result indices Wim Pomp 2024-09-05 18:37:47 +02:00
  • 29203dd128 - redirect stdout and stderr in child processes to devnull instead of None - add parfor.__version__ Wim Pomp 2024-07-22 17:32:17 +02:00
  • 9ce96a7e61 - redirect stdout and stderr in child processes Wim Pomp 2024-07-08 17:57:23 +02:00
  • b609df8b52 - bugfix: do not restart pool when n_processes = None Wim Pomp 2024-05-28 16:35:14 +02:00
  • 9783c1d1f2 - introduce n_processes to change the number of processes in the pool Wim Pomp 2024-05-24 16:57:35 +02:00
  • ac4d599646 - minimum python: 3.10 - typing - task_error bug fix - remove some deprecated functions Wim Pomp 2024-04-26 18:32:12 +02:00
  • 42746d21eb - make ParPoolSingleton.close a class method. Wim Pomp 2024-03-19 18:52:20 +01:00
  • 6fefdb9be0 - bugfix for argument bar=False Wim Pomp 2024-02-09 13:32:18 +01:00
  • bb3b87b913 - catch and ignore errors when emptying queue - timeout on closing pool - close manager on closing pool Wim Pomp 2024-01-29 11:25:22 +01:00
  • 0b0c7a798f - Shutdown workers after 10 minutes of inactivity. Restart pool when needed again. Wim Pomp 2023-11-18 13:38:01 +01:00
  • ad8d9a4efb - Protect reference counter with a lock because changes to a multiprocessing manager dict are not atomic. Wim Pomp 2023-11-08 12:50:30 +01:00
  • 098d1810c3 - Bugfix Wim Pomp 2023-11-07 16:51:02 +01:00
  • f22b682b2a - skip KeyboardInterrupts in workers - silence some warnings Wim Pomp 2023-11-07 15:04:54 +01:00
  • e936292905 - Implement reference counting to make sure the parent process does not trash the item before all children trashed it. Wim Pomp 2023-10-27 17:48:52 +02:00
  • 265470e0ac - Use a shared memory approach. - Track unpickled objects by CouldNotBePickled class. - Bump minimal python to 3.8. Wim Pomp 2023-10-12 14:46:42 +02:00
  • 0263b6a4c7 - Copy previous task to prevent unnecessary pickling. Wim Pomp 2023-09-13 17:04:55 +02:00
  • f3302f6dba - Use a singleton pool to prevent lengthy restarts of the pool, this also means that arguments for pool size have gone. - Removed qbar and TqdmMeter. - Wrap chunked function for better error messages. Wim Pomp 2023-09-08 17:19:35 +02:00
  • 92f162b7d5 - Make processes non-daemonic - Deprecate terminator argument Wim Pomp 2023-08-16 16:46:50 +02:00
  • dfee4bc0ea - README update Wim Pomp 2023-08-08 17:50:20 +02:00
  • 0635c62d42 - bugfix: some iterables caused an error - pytest tests - deprecation warnings - use poetry for installs Wim Pomp 2023-08-08 15:05:31 +02:00
  • 7e0c0cc45d - bugfix in parpool.get_newest - make TqdmMeter bar decrease when necessary Wim Pomp 2022-06-28 11:08:37 +02:00
  • 84f4afbba2 Merge remote-tracking branch 'origin/master' Wim Pomp 2022-05-03 15:56:20 +02:00
  • 32c5560306 - rename some things to conform to PEP - pmap and parfor now automatically divide the task in chunks: less pickling is needed this way - pickle code in a new file: prevent some unpickleble things from getting pickled - Chunks now supports generators when the length keyword argument is supplied - TqdmMeter can have a description Wim Pomp 2022-05-03 15:55:11 +02:00
  • d2c226a37e - rename some things to conform to PEP - pmap and parfor now automatically divide the task in chunks: less pickling is needed this way - pickle code in a new file: prevent some unpickleble things from getting pickled - Chunks now supports generators when the length keyword argument is supplied - TqdmMeter can have a description Wim Pomp 2022-05-03 15:55:11 +02:00
  • 865ec70d97 - bug fix in killed process detection Wim Pomp 2022-03-22 15:36:07 +01:00
  • fcf451cb23 - drop support for python 2 - automatically kill javabridge (if it exists) at the end of a process - better hashing and tasking Wim Pomp 2022-03-03 12:43:00 +01:00
  • 4bddca82eb - option for a callback function to update a user defined bar Wim Pomp 2021-07-29 21:04:28 +02:00
  • 8ed1a5500e - chunks: kwargs to set number of chunks relative to number of cpus Wim Pomp 2021-07-17 10:54:00 +02:00
  • bc2ae2ea25 - Make progress bar update with chunksize when using chunks. - Catch exceptions, but not keyboard interrupts. Wim Pomp 2021-05-29 11:09:54 +02:00
  • 9caeb06c2b - Bugfix Wim Pomp 2021-05-25 10:51:14 +02:00
  • 6c9c18746b - Make chunks a class, so we can do len(chunks(...)) Wim Pomp 2021-05-24 18:59:32 +02:00
  • 95ae325028 - Better chunks function, option for both number of chunks or size of chunks. Wim Pomp 2021-05-24 18:19:14 +02:00
  • 5d57081713 - Better and faster error handling. Wim Pomp 2021-03-19 20:39:38 +01:00
  • ddc0189bad - Add get_newest method to parpool. Wim Pomp 2021-03-06 00:09:16 +01:00
  • 6cce9e9b8e - Better error handling Wim Pomp 2021-03-05 22:27:00 +01:00
  • 61f1429543 - bar for serial case Wim Pomp 2021-02-20 15:17:17 +01:00
  • 3d99250b98 - force number of workers to be at least 2 - update readme Wim Pomp 2021-01-04 12:35:01 +01:00
  • 0721e2125c - differentiate use of nP into nP and rP - default to using one process per cpu core - update requirements with versions Wim Pomp 2021-01-04 12:25:19 +01:00
  • b979a0c72a Merge remote-tracking branch 'origin/master' Wim Pomp 2020-12-28 13:53:36 +01:00
  • d7a2a473d5 Update readme with if __name__ == '__main__': Wim Pomp 2020-12-28 13:52:05 +01:00
  • a8fefb5d00 Update readme with if __name__ == '__main__': Wim Pomp 2020-12-28 13:52:05 +01:00
  • a02677dc1d - Fix chunks for when length of input is smaller than chunk size. Wim Pomp 2020-11-24 16:34:40 +01:00
  • 2ddf73ed0f - Fix qbar error for OSX. w.pomp 2020-09-11 17:37:47 +02:00
  • 6c4f2d5132 - Add terminator function. w.pomp 2020-09-04 18:03:30 +02:00
  • da70cf7a2f - Dill (almost) everything. - Change dill a little to automatically omit undillable parts of objects. - README.me Limitations w.pomp 2020-09-03 11:29:15 +02:00
  • 881496b8f4 - Add github link to setup.py - Add install_requires w.pomp 2020-08-31 17:39:52 +02:00
  • 0d07638e98 First commit. w.pomp 2020-08-31 16:10:12 +02:00