多进程命令列表

Question

我试图运行一个命令列表。问题是列表可能很长，所以同时运行多个命令会很好。

如何在多处理模块中执行此操作？

list_of_commands = [cmd foo, cmd bar, ...] 

main_log_file = open(os.getcwd() + '/Error.log', 'w+') 

Count = 0 
for Job in list_of_commands: 
    Count += 1 
    child = subprocess.Popen(Job, stdout=subprocess.PIPE, stderr=subprocess.PIPE) 
    streamdata = child.communicate()[0] 
    errcode = child.returncode 
    if errcode == 0: 
     print ('Job', Count, 'Success') 
    elif errcode == 1: 
     print ('Job', Count, 'Completed With Errors') 
    elif errcode == 2: 
     print ('Job', Count, 'Error') 
     main_log_file.write (streamdata.decode('ascii') + str(errcode) + '\n') 

main_log_file.close() 

执行的顺序并不重要

Answer 1

可以使用concurrent.futures.ThreadPoolExecutormap功能以运行并行执行的costant数。

WORKERS = 5 # amount of concurrent executions you want 

def launcher(job): 
    child = subprocess.Popen(job, ...) 
    streamdata = child.communicate()[0] 
    ... 

with concurrent.futures.ThreadPoolExecutor(max_workers=WORKERS) as pool: 
    pool.map(launcher, jobs)