如何在AppEngine MapReduce中获得计数器结果？

Question

我正在使用google mapreduce lib来处理我的数据。在处理数据时，计数器用于映射器功能。但我不知道如何通过最终方法获得反击结果。

def mapper(obj): 
    yield obj 
    yield operation.counters.Increment("process-obj") 


class Test(base_handler.PipelineBase): 
    """A pipeline to ingest log as CSV in Google Storage 
    """ 

    def run(self, setting_id): 
     filepath = yield mapreduce_pipeline.MapperPipeline(
      "test", 
      "mapper", 
      "mapreduce.input_readers.DatastoreInputReader", 
      output_writer_spec="mapreduce.output_writers.FileOutputWriter", 
      params={ 

      }, 
      shards=10 
     ) 
    def finalized(self): 
     # how to read the counter process-obj 
     # how to get the setting_id 
     pass 

Answer 1

命名输出可能是你在找什么。你可以找到更多的细节here。

下面是使用命名输出拿回各种计数器，包括您所定义的一个代码：

def mapper(obj): 
    yield obj 
    yield operation.counters.Increment("process-obj") 


class Test(base_handler.PipelineBase): 
    """A pipeline to ingest log as CSV in Google Storage 
    """ 

    output_names = ['counters'] 

    def run(self, setting_id): 
     results = yield mapreduce_pipeline.MapperPipeline(
      "test", 
      "mapper", 
      "mapreduce.input_readers.DatastoreInputReader", 
      output_writer_spec="mapreduce.output_writers.FileOutputWriter", 
      params={ 

      }, 
      shards=10 
     ) 
     yield MapreduceResult(results.counters) 

    def finalized(self): 
     print 'Counters here: ', self.outputs.counters 


class MapreduceResult(base_handler.PipelineBase): 

    def run(self, counters): 
     self.fill(self.outputs.counters, counters)