Software
pymizu

pymizu

Async and parallel Python at thread speed over shared memory

pymizu is a Python package for async and parallel computing that communicates over shared memory instead of sockets. A task round trip drops from ~100 µs over sockets to under 1 µs, so work can be divided as finely as it would be across threads.

pymizu is built on libmizu, a C library for lock-free shared-memory IPC, and mizu binds the same core for R, so pool workers can be R processes and lists, dicts, NumPy arrays, and Arrow tables cross between Python and R as zero-copy views. Pools provide a drop-in concurrent.futures.Executor with real Future objects, and tasks, maps, and channels can be awaited directly in asyncio without polling threads. Parallel random numbers are reproducible for any worker count, and a task can map over its own pool without deadlock.

Contributors#

Posts about pymizu#