将地图缩减作业的输出记录到文本文件中

Question

我一直使用此jobclient.monitorandprintjob（）方法将地图缩小作业的输出打印到控制台。我的用法是这样的：

job_client.monitorAndPrintJob(job_conf, job_client.getJob(j.getAssignedJobID())) 

它的输出如下（印在控制台上）：

13/03/04 07:20:00 INFO mapred.JobClient: Running job: job_201302211725_10139<br> 
13/03/04 07:20:01 INFO mapred.JobClient: map 0% reduce 0%<br> 
13/03/04 07:20:08 INFO mapred.JobClient: map 100% reduce 0%<br> 
13/03/04 07:20:13 INFO mapred.JobClient: map 100% reduce 100%<br> 
13/03/04 07:20:13 INFO mapred.JobClient: Job complete: job_201302211725_10139<br> 
13/03/04 07:20:13 INFO mapred.JobClient: Counters: 26<br> 
13/03/04 07:20:13 INFO mapred.JobClient: Job Counters<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Launched reduce tasks=1<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Aggregate execution time of mappers(ms)=5539<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Total time spent by all reduces waiting after reserving slots (ms)=0<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Total time spent by all maps waiting after reserving slots (ms)=0<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Launched map tasks=2<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Data-local map tasks=2<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Aggregate execution time of reducers(ms)=4337<br> 
13/03/04 07:20:13 INFO mapred.JobClient: FileSystemCounters<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  MAPRFS_BYTES_READ=583<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  MAPRFS_BYTES_WRITTEN=394<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  FILE_BYTES_WRITTEN=140219<br> 
13/03/04 07:20:13 INFO mapred.JobClient: Map-Reduce Framework<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Map input records=6<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Reduce shuffle bytes=136<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Spilled Records=22<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Map output bytes=116<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  CPU_MILLISECONDS=1320<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Map input bytes=64<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Combine input records=13<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  SPLIT_RAW_BYTES=180<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Reduce input records=11<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Reduce input groups=11<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Combine output records=11<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  PHYSICAL_MEMORY_BYTES=734961664<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Reduce output records=11<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  VIRTUAL_MEMORY_BYTES=9751805952<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  Map output records=13<br> 
13/03/04 07:20:13 INFO mapred.JobClient:  GC time elapsed (ms)=0<br> 

我想上面的输出/日志在一个文本文件要打印，而不是控制台。有什么建议么？

Answer 1

在您的HADOOP_HOME/conf中，您可能会发现一个文件名为：log4j.properties。我相信你可以配置在哪里以及如何登录。

准确地说，你必须使用一个滚动文件附加目的地，所以你应当取消注释（只是删除#）从log4j.properties文件中以下行：

# Rolling File Appender 
# 

#log4j.appender.RFA=org.apache.log4j.RollingFileAppender 
#log4j.appender.RFA.File=${hadoop.log.dir}/${hadoop.log.file} 

# Logfile size and and 30-day backups 
#log4j.appender.RFA.MaxFileSize=1MB 
#log4j.appender.RFA.MaxBackupIndex=30 

#log4j.appender.RFA.layout=org.apache.log4j.PatternLayout 
#log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} - %m%n 
#log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n 

和自定义其他参数你喜欢。

有关log4j配置的更多信息，请参阅here。