Hadoop 0.23.9如何启动datanodes

Question

看来我无法让hadoop正常启动。我使用Hadoop 0.23.9：

[msknapp@localhost sbin]$ hadoop namenode -format 
... 
[msknapp@localhost sbin]$ ./start-dfs.sh 
Starting namenodes on [localhost] 
localhost: starting namenode, logging to /usr/local/cloud/hadoop-0.23.9/logs/hadoop-msknapp-namenode-localhost.localdomain.out 
localhost: starting datanode, logging to /usr/local/cloud/hadoop-0.23.9/logs/hadoop-msknapp-datanode-localhost.localdomain.out 
Starting secondary namenodes [0.0.0.0] 
0.0.0.0: starting secondarynamenode, logging to /usr/local/cloud/hadoop-0.23.9/logs/hadoop-msknapp-secondarynamenode-localhost.localdomain.out 
[msknapp@localhost sbin]$ ./start-yarn.sh 
starting yarn daemons 
starting resourcemanager, logging to /usr/local/cloud/hadoop-0.23.9/logs/yarn-msknapp-resourcemanager-localhost.localdomain.out 
localhost: starting nodemanager, logging to /usr/local/cloud/hadoop-0.23.9/logs/yarn-msknapp-nodemanager-localhost.localdomain.out 
[msknapp@localhost sbin]$ cd /var/local/stock/data 
[msknapp@localhost data]$ hadoop fs -ls/
[msknapp@localhost data]$ hadoop fs -mkdir /stock 
[msknapp@localhost data]$ ls 
companies.csv raw slf_series.txt 
[msknapp@localhost data]$ hadoop fs -put companies.csv /stock/companies.csv 
13/12/08 11:10:40 WARN hdfs.DFSClient: DataStreamer Exception 
java.io.IOException: File /stock/companies.csv._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. 
    at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1180) 
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1536) 
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:414) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:606) 
    at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:394) 
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1571) 
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1567) 
    at java.security.AccessController.doPrivileged(Native Method) 
    at javax.security.auth.Subject.doAs(Subject.java:415) 
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1262) 
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1565) 

    at org.apache.hadoop.ipc.Client.call(Client.java:1094) 
    at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:195) 
    at com.sun.proxy.$Proxy6.addBlock(Unknown Source) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:606) 
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:102) 
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:67) 
    at com.sun.proxy.$Proxy6.addBlock(Unknown Source) 
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1130) 
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1006) 
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:458) 
put: File /stock/companies.csv._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. 
13/12/08 11:10:40 ERROR hdfs.DFSClient: Failed to close file /stock/companies.csv._COPYING_ 
java.io.IOException: File /stock/companies.csv._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. 
    at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1180) 
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1536) 
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:414) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:606) 
    at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:394) 
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1571) 
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1567) 
    at java.security.AccessController.doPrivileged(Native Method) 
    at javax.security.auth.Subject.doAs(Subject.java:415) 
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1262) 
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1565) 

    at org.apache.hadoop.ipc.Client.call(Client.java:1094) 
    at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:195) 
    at com.sun.proxy.$Proxy6.addBlock(Unknown Source) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:606) 
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:102) 
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:67) 
    at com.sun.proxy.$Proxy6.addBlock(Unknown Source) 
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1130) 
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1006) 
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:458) 

这里是我的核心-site.xml中：

<property> 
    <name>fs.default.name</name> 
    <value>hdfs://localhost/</value> 
</property> 

和我的HDFS-site.xml中：

<property> 
     <name>dfs.replication</name> 
     <value>1</value> 
</property> 

和mapred- site.xml：

<property> 
      <name>mapred.job.tracker</name> 
      <value>localhost:8021</value> 
    </property> 

我查看了所有文档我不知道如何正确启动hadoop。我在网上找不到关于hadoop-0.23.9的任何文档。我的Hadoop书是为0.22编写的。在线文档适用于2.1.1，这恰巧我无法工作。

有人可以告诉我如何让我的hadoop开始正确吗？

Answer 1

指定fs.default.name

像一个端口：

<property> 
    <name>fs.default.name</name> 
    <value>hdfs://localhost:54310</value> 
</property> 

之后，创建一个tmp目录为HDFS：

sudo mkdir -p /app/hadoop/tmp 
sudo chown you /app/hadoop/tmp 

，并加入到核心的site.xml：

<property> 
    <name>hadoop.tmp.dir</name> 
    <value>/app/hadoop/tmp</value> 
    <description>A base for other temporary directories.</description> 
</property> 

确保重新启动群集。

$HADOOP_HOME/bin/stop-all.sh 
$HADOOP_HOME/bin/start-all.sh 

Answer 2

尝试使用Hadoop删除所有数据停止：

$HADOOP_HOME/bin/hadoop datanode -format 

或手动删除

/应用/ Hadoop的内容的/ tmp/DFS /数据/

然后再次启动hadoop：

$HADOOP_HOME/bin/start-all.sh 

Answer 3

配置中的关键问题是如下：

java.io.IOException: File /stock/companies.csv._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. 

确保您的HDFS具体配置至少有最低的，如下：

HDFS-site.xml中： 作为节目的xml你必须有/ tmp/hdfs23/namenode和/ tmp/hdfs23/datanode文件夹已经存在。你可以为hdfs root配置任何其他文件夹，然后配置namenode和datanode文件夹。

<configuration> 
     <property> 
      <name>dfs.replication</name> 
      <value>1</value> 
     </property> 
     <property> 
      <name>dfs.namenode.name.dir</name> 
      <value>file:///tmp/hdfs23/namenode</value> 
     </property> 
     <property> 
      <name>fs.checkpoint.dir</name> 
      <value>file:///tmp/hdfs23/secnamenode</value> 
     </property> 
     <property> 
      <name>fs.checkpoint.edits.dir</name> 
      <value>file:///tmp/hdfs23/secnamenode</value> 
     </property> 
     <property> 
      <name>dfs.datanode.data.dir</name> 
      <value>file:///tmp/hdfs23/datanode</value> 
     </property> 
</configuration> 

核心网站。XML

<configuration> 
     <property> 
      <name>fs.defaultFS</name> 
      <value>hdfs://localhost:9000</value> 
     </property> 
    <property> 
     <name>fs.default.name</name> 
     <value>hdfs://localhost:9000</value> 
    </property> 
    <property> 
     <name>hadoop.http.staticuser.user</name> 
     <value>hdfs</value> 
    </property> 
</configuration> 

然后，你需要格式化你的NameNode因为你已经做了：

$ hadoop namenode -format 

之后，你可以如下启动HDFS：

[Hadoop023_ROOT]/sbin/start-dfs.sh