删除列表中的对象以传递给多处理

Question

我需要将大列表中的每个对象传递给一个函数。函数完成后，我不再需要传递给该函数的对象，并希望删除该对象以节省内存。如果我用一个单一的过程中工作，我将做到以下几点：

result = [] 
while len(mylist) > 0: 
    result.append(myfunc(mylist.pop()) 

正如我遍历MYLIST它传递给我的功能后，我突然掉在这样的对象不再存储在MYLIST列表中的每个对象。如何使用multiprocessing并行实现同样的效果？

Answer 1

一个简单的消费者例如（credits go here）：

import multiprocessing 
import time 
import random 

class Consumer(multiprocessing.Process): 

    def __init__(self, task_queue, result_queue): 
     multiprocessing.Process.__init__(self) 
     self.task_queue = task_queue 
     self.result_queue = result_queue 

    def run(self): 
     while True: 
      task = self.task_queue.get() 
      if task is None: 
       # Poison pill means shutdown 
       self.task_queue.task_done() 
       break 
      answer = task.process() 
      self.task_queue.task_done() 
      self.result_queue.put(answer) 
     return 


class Task(object): 

    def process(self): 
     time.sleep(0.1) # pretend to take some time to do the work 
     return random.randint(0, 100) 


if __name__ == '__main__': 
    # Establish communication queues 
    tasks = multiprocessing.JoinableQueue() 
    results = multiprocessing.Queue() 

    # Start consumers 
    num_consumers = multiprocessing.cpu_count() * 2 
    consumers = [Consumer(tasks, results) for i in xrange(num_consumers)] 
    for consumer in consumers: 
     consumer.start() 

    # Enqueue jobs 
    num_jobs = 10 
    for _ in xrange(num_jobs): 
     tasks.put(Task()) 

    # Add a poison pill for each consumer 
    for _ in xrange(num_consumers): 
     tasks.put(None) 

    # Wait for all tasks to finish 
    tasks.join() 

    # Start printing results 
    while num_jobs: 
     result = results.get() 
     print 'Result:', result 
     num_jobs -= 1