Distributed Computing (🖥️ X10000)

Namespaces don’t work: ## Both the functions would fail to pickle, ## instead just have a blank function in a module. class Namespace: @staticmethod def test_static(): return "Finished" def test(): return "Finished" ## This might work exec(open("code/Trappy-Scopes/trackyscope/splitprocessor.py", "r").read()) Imports need to happen inside the function, also PATH list of each function is independent and needs to be redefined: def process_func(): import sys sys.path.appen("path-to-your-module") ## Calculations return ## Won't work import time def process_func(): time.sleep(2) return "Finished" ## Works def process_func(): import time time.sleep(2) return "Finished" cell1.client.get_versions(check=True) ...

Introduction We will set-up a cluster using a job-scheduler (SLURM) and use dask-distributed to manage jobs. One does not normally choose the job-scheduler, it is given to us (like your institute or HPC service would use a specific one). It is for this reason that dask comes in handy. It implements a trivial-parallelisation model on top of most common job-schedulers (SLURM, PBS, etc) to name a few. The interface it provides resembles python’s multiprocessing module. ...