multiprocessing - Multiprocess a list of commands -
im trying run list of commands. problem list can long, having multiple commands run simultaneously great.
how do multiprocessing module?
list_of_commands = [cmd foo, cmd bar, ...] main_log_file = open( os.getcwd() + '/error.log', 'w+') count = 0 job in list_of_commands: count += 1 child = subprocess.popen(job, stdout=subprocess.pipe, stderr=subprocess.pipe) streamdata = child.communicate()[0] errcode = child.returncode if errcode == 0: print ( 'job', count, 'success' ) elif errcode == 1: print ( 'job', count, 'completed errors' ) elif errcode == 2: print ( 'job', count, 'error' ) main_log_file.write ( streamdata.decode('ascii') + str(errcode) + '\n' ) main_log_file.close()
order of execution doesn't matter
you can use concurrent.futures.threadpoolexecutor map
function run costant number of parallel executions.
workers = 5 # amount of concurrent executions want def launcher(job): child = subprocess.popen(job, ... ) streamdata = child.communicate()[0] ... concurrent.futures.threadpoolexecutor(max_workers=workers) pool: pool.map(launcher, jobs)
Comments
Post a Comment