Parallelization with Dask¶
[1]:
from dask.distributed import LocalCluster, Client
cluster = LocalCluster()
client = Client(cluster)
client
[1]:
Client
Client-f4f33458-7235-11f0-bc00-be1a4b830aa9
| Connection method: Cluster object | Cluster type: distributed.LocalCluster |
| Dashboard: http://127.0.0.1:8787/status |
Cluster Info
LocalCluster
7b390fd6
| Dashboard: http://127.0.0.1:8787/status | Workers: 7 |
| Total threads: 14 | Total memory: 36.00 GiB |
| Status: running | Using processes: True |
Scheduler Info
Scheduler
Scheduler-cdf8695e-51e9-4855-89ba-1bdb3c366c6d
| Comm: tcp://127.0.0.1:53361 | Workers: 0 |
| Dashboard: http://127.0.0.1:8787/status | Total threads: 0 |
| Started: Just now | Total memory: 0 B |
Workers
Worker: 0
| Comm: tcp://127.0.0.1:53378 | Total threads: 2 |
| Dashboard: http://127.0.0.1:53382/status | Memory: 5.14 GiB |
| Nanny: tcp://127.0.0.1:53364 | |
| Local directory: /var/folders/8f/1b9jykm565z0yjl1_pkz3tt40000gn/T/dask-scratch-space/worker-0a7ek597 | |
Worker: 1
| Comm: tcp://127.0.0.1:53380 | Total threads: 2 |
| Dashboard: http://127.0.0.1:53386/status | Memory: 5.14 GiB |
| Nanny: tcp://127.0.0.1:53366 | |
| Local directory: /var/folders/8f/1b9jykm565z0yjl1_pkz3tt40000gn/T/dask-scratch-space/worker-rhhm6tl3 | |
Worker: 2
| Comm: tcp://127.0.0.1:53379 | Total threads: 2 |
| Dashboard: http://127.0.0.1:53383/status | Memory: 5.14 GiB |
| Nanny: tcp://127.0.0.1:53368 | |
| Local directory: /var/folders/8f/1b9jykm565z0yjl1_pkz3tt40000gn/T/dask-scratch-space/worker-89muf7s4 | |
Worker: 3
| Comm: tcp://127.0.0.1:53381 | Total threads: 2 |
| Dashboard: http://127.0.0.1:53389/status | Memory: 5.14 GiB |
| Nanny: tcp://127.0.0.1:53370 | |
| Local directory: /var/folders/8f/1b9jykm565z0yjl1_pkz3tt40000gn/T/dask-scratch-space/worker-cvo9miq6 | |
Worker: 4
| Comm: tcp://127.0.0.1:53388 | Total threads: 2 |
| Dashboard: http://127.0.0.1:53392/status | Memory: 5.14 GiB |
| Nanny: tcp://127.0.0.1:53372 | |
| Local directory: /var/folders/8f/1b9jykm565z0yjl1_pkz3tt40000gn/T/dask-scratch-space/worker-c9o_6x_l | |
Worker: 5
| Comm: tcp://127.0.0.1:53391 | Total threads: 2 |
| Dashboard: http://127.0.0.1:53395/status | Memory: 5.14 GiB |
| Nanny: tcp://127.0.0.1:53374 | |
| Local directory: /var/folders/8f/1b9jykm565z0yjl1_pkz3tt40000gn/T/dask-scratch-space/worker-m5g8sgks | |
Worker: 6
| Comm: tcp://127.0.0.1:53394 | Total threads: 2 |
| Dashboard: http://127.0.0.1:53397/status | Memory: 5.14 GiB |
| Nanny: tcp://127.0.0.1:53376 | |
| Local directory: /var/folders/8f/1b9jykm565z0yjl1_pkz3tt40000gn/T/dask-scratch-space/worker-x8a3xu7e | |
[2]:
client.close()