通过Java API读取HDFS

[可能的重复]我正在尝试使用Java API读取HDFS。使用命令行和Hadoop Url它工作正常但问题是阅读而hdfs路径。我经历了这Reading HDFS and local files in Java，但我无法找到我错在哪里。通过Java API读取HDFS

1）命令行给出了这样的结果

[email protected]:~$ hadoop fs -ls 
    Found 3 items 
    drwxr-xr-x - hduser supergroup   0 2014-01-11 00:21 /user/hduser/In 
    -rw-r--r-- 1 hduser supergroup 37461150 2014-01-11 17:27 /user/hduser/loging.txt 
    -rw-r--r-- 3 hduser supergroup 112383446 2014-01-11 19:02 /user/hduser/loging1.txt

2）

 static { 
       URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory()); 
      } 
      InputStream in = null; 
    try { 
*********************This is working fine******************************** 
     in = new URL("hdfs://localhost:54310/user/hduser/loging.txt") 
       .openStream(); 
************************************************************************* 

*************************This is not working***************************** 
     in = new URL("hdfs://localhost/user/hduser/loging.txt").openStream(); 

     It says: 
     14/01/11 19:54:55 INFO ipc.Client: Retrying connect to server: 
     localhost/127.0.0.1:8020. Already tried 0 time(s). 
           . 
           . 
     14/01/11 19:55:04 INFO ipc.Client: Retrying connect to server:  
     localhost/127.0.0.1:8020. Already tried 9 time(s). 
     Exception in thread "main" java.net.ConnectException: Call to  
     localhost/127.0.0.1:8020 failed on connection exception: java.net.ConnectException: Connection refused 
**************************************************************************  

     IOUtils.copyBytes(in, System.out, 4096, false); 

    } finally { 
     IOUtils.closeStream(in); 
    }

3）给予例外

Configuration configuration = new Configuration(); 
    configuration.addResource(new Path("/hadoop/conf/core-site.xml")); 
    configuration.addResource(new Path("/hadoop/conf/hdfs-site.xml")); 

    FileSystem fileSystem = FileSystem.get(configuration); 
    System.out.println(fileSystem.getHomeDirectory()); 
    Path path = new Path("/user/hduser/loging.txt"); 

    FSDataInputStream in = fileSystem.open(path); 
    System.out.println(in); 
    byte[] b = new byte[1024]; 
    int numBytes = 0; 
    while ((numBytes = in.read(b)) > 0) { 
     //processing 
    } 

    in.close(); 

    fileSystem.close(); 
Exception: 
file:/home/hduser 
Exception in thread "main" java.io.FileNotFoundException: File /user/hduser/loging.txt does not exist. 
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361) 
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245) 
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125) 
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283) 
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:356) 
at HdfsUrl.read(HdfsUrl.java:29) 
at HdfsUrl.main(HdfsUrl.java:58)

来源

2014-01-11 dinesh

那种你所得到的异常表明这就是你的应用程序试图连接到没有打开8020端口的datanode。按照此documentation 8020是默认端口namenode。我建议添加你的主机名和端口信息。在你的core-site.xml。像这样：

<property> 
    <name>fs.default.name</name> 
    <value>hdfs://[namenode]:[port]</value> 
</property>

来源

2014-01-11 15:22:32 Ashish

通过Java API读取HDFS

回答

相关问题