运行Python代码Hadoop的失败

Question

我试图按照此页面上的说明： http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/

$bin/hadoop jar contrib/streaming/hadoop-streaming-1.0.4.jar -input /user/root/wordcountpythontxt -output /user/root/wordcountpythontxt-output -mapper /user/root/wordcountpython/mapper.py -reducer /user/root/wordcountpython/reducer.py -file /user/root/mapper.py -file /user/root/reducer.py 

它说

File: /user/root/mapper.py does not exist, or is not readable. 
Streaming Command Fail 

当我通过URL浏览：jobdetails.jsp/

我发现很多异常

java.lang.RuntimeException: Error in configuring object 
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93) 
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) 
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) 
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432) 
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) 
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255) 
    at java.security.AccessController.doPrivileged(Native Method) 
    at javax.security.auth.Subject.doAs(Subject.java:396) 
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) 
    at org.apache.hadoop.mapred.Child.main(Child.java:249) 
Caused by: java.lang.reflect.InvocationTargetException 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
    at java.lang.reflect.Method.invoke(Method.java:597) 
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) 
    ... 9 more 
Caused by: java.lang.RuntimeException: Error in configuring object 
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93) 
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) 
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) 
    at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34) 
    ... 14 more 
Caused by: java.lang.reflect.InvocationTargetException 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
    at java.lang.reflect.Method.invoke(Method.java:597) 
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) 
    ... 17 more 
Caused by: java.lang.RuntimeException: configuration exception 
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:230) 
    at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66) 
    ... 22 more 
Caused by: java.io.IOException: Cannot run program "/user/root/wordcountpython/mapper.py": error=2, No such file or directory 
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:460) 
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:214) 
    ... 23 more 
Caused by: java.io.IOException: error=2, No such file or directory 
    at java.lang.UNIXProcess.forkAndExec(Native Method) 
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:53) 
    at java.lang.ProcessImpl.start(ProcessImpl.java:91) 
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:453) 
    ... 24 more 

我不能修复它请帮我运行python pgm。

Answer 1

如果你仔细检查了link的指示，

hduser@ubuntu:/usr/local/hadoop$ bin/hadoop jar contrib/streaming/hadoop-*streaming*.jar -file /home/hduser/mapper.py -mapper /home/hduser/mapper.py -file /home/hduser/reducer.py -reducer /home/hduser/reducer.py -input /user/hduser/gutenberg/* -output /user/hduser/gutenberg-output 

有它清楚地表明没有必要复制mapper.py和reducer.py到HDFS，你可以将文件从两个链接本地文件系统：as/path/to/mapper。我相信你可以避免上述错误。

Answer 2

看来问题是符合的。

Caused by: java.io.IOException: Cannot run program "/user/root/wordcountpython/mapper.py": error=2, No such file or directory 

请您检查文件/user/root/wordcountpython/mapper.py是否存在。如果它存在，那么该文件的权限是什么。

您正在运行的用户hadoop有权限执行和读取此文件？

Answer 3

您可能想要检查您的＃后没有dos样式的新行！在mapper.py中行。如果你这样做，hadoop可能无法找到你的python解释器，因为它会看到一个额外的CR。例如。/usr/local/bin/python^M而不是/ usr/local/bin/python其中^ M是CR。在你的mapper和reducer上尝试dos2unix命令。