我试过这个scala代码从ftp站点下载文件。代码如下。如何用SparkContext.textFile ftp文件?
object BasicTextFromFTP {
def main(args: Array[String]) {
val conf = new org.apache.spark.SparkConf().setAppName("FTP Test")
conf.setMaster("local")
val sc = new SparkContext(conf)
val file = sc.textFile("ftp://anonymous:[email protected]/ubuntu/ls-LR.gz")
println(file.collect().mkString("\n"))
}
}
当我运行它时出现以下错误。
16/02/12 10时52分22秒INFO SparkContext:在从BasicTextFromFTP.scala TEXTFILE创建广播0:在螺纹14 异常 “主” org.apache.hadoop.mapred.InvalidInputException:输入路径确实不存在:ftp://anonymous:[email protected]/ubuntu/ls-LR.gz 在org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:251) 在org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:270) 在org.apache。 spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:199) at org.apache.spark.rdd.RDD $$ anonfun $ partitions $ 2.apply(RDD.scala:239) at org.apache.spark.rdd。 RDD $$ anonfun $ partitions $ 2.apply(RDD.scala:237) at org.apache.spark.rdd.RDD.partitions(RDD.scala:237) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD。)中的scala.Option.getOrElse(Option.scala:121) scala:35) at org.apache.spark.rdd.RDD $$ anonfun $ partitions $ 2.apply(RDD.scala:239) at org.apache.spark.rdd.RDD $$ anonfun $ partitions $ 2.apply( RDD.scala:237) at org.apache.spark.rdd.RDD.partitions(RDD.scala:237) at org.apache.spark.SparkContext。scala.Option.getOrElse(Option.scala:121) 运行作业(SparkContext.scala:1929) at org.apache.spark.rdd.RDD $$ anonfun $ collect $ 1.apply(RDD.scala:927) at org.apache.spark.rdd.RDDOperationScope $ .withScope(RDDOperationScope .scala:150) at org.apache.spark.rdd.R DDOperationScope $ .withScope(RDDOperationScope.scala:111) at org.apache.spark.rdd.RDD.withScope(RDD.scala:316) at org.apache.spark.rdd.RDD.collect(RDD.scala:926 ) 在ftp.BasicTextFromFTP $。主要(BasicTextFromFTP.scala:15) 在ftp.BasicTextFromFTP.main(BasicTextFromFTP.scala) 在sun.reflect.NativeMethodAccessorImpl.invoke0(本机方法) 在sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.intellij.rt.execution .application.AppMain.main(AppMain.java:144)
我使用spark 1.6.0和scala 2.11。
我已经尝试手动ftp和filezilla之前,我发布此问题。两种方式都有效。 –