2017-04-01 23 views
0

我已经创建了一个azure hdinsight火花集群,我试图在headnode上安装Spark Job Server。我已经ssh'ed到headnode,这里是我下面火花作业服务器安装在hdisght

其HDI 3.5星火1.6.3

  1. 的步骤安装SBT:

回声 “的deb https://dl.bintray.com/sbt/debian /” |须藤发球-a /etc/apt/sources.list.d/sbt.list

sudo易于关键进阶--keyserver HKP://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823

须藤易于得到更新

须藤易于得到安装SBT

  • GIT中克隆https://github.com/spark-jobserver/spark-jobserver.git
  • 主是用于火花1.6.3,因此我不改变b牧场。

  • 移动到火花观察者目录

  • SBT组件 此时我正在错误这样

  • *

    [warn] Merging 'reference.conf' with strategy 'concat' 
    [error] 1 error was encountered during merge 
    java.lang.RuntimeException: deduplicate: different file contents found in the following: 
    /home/kmk/.ivy2/cache/io.netty/netty-all/jars/netty-all-4.0.37.Final.jar:META-INF/io.netty.versions.properties 
    /home/kmk/.ivy2/cache/io.netty/netty-handler/jars/netty-handler-4.0.37.Final.jar:META-INF/io.netty.versions.properties 
    /home/kmk/.ivy2/cache/io.netty/netty-buffer/jars/netty-buffer-4.0.37.Final.jar:META-INF/io.netty.versions.properties 
    /home/kmk/.ivy2/cache/io.netty/netty-common/jars/netty-common-4.0.37.Final.jar:META-INF/io.netty.versions.properties 
    /home/kmk/.ivy2/cache/io.netty/netty-transport/jars/netty-transport-4.0.37.Final.jar:META-INF/io.netty.versions.properties 
    /home/kmk/.ivy2/cache/io.netty/netty-codec/jars/netty-codec-4.0.37.Final.jar:META-INF/io.netty.versions.properties 
        at sbtassembly.Assembly$.applyStrategies(Assembly.scala:140) 
        at sbtassembly.Assembly$.x$1$lzycompute$1(Assembly.scala:25) 
        at sbtassembly.Assembly$.x$1$1(Assembly.scala:23) 
        at sbtassembly.Assembly$.stratMapping$lzycompute$1(Assembly.scala:23) 
        at sbtassembly.Assembly$.stratMapping$1(Assembly.scala:23) 
        at sbtassembly.Assembly$.inputs$lzycompute$1(Assembly.scala:67) 
        at sbtassembly.Assembly$.inputs$1(Assembly.scala:57) 
        at sbtassembly.Assembly$.apply(Assembly.scala:83) 
        at sbtassembly.Assembly$$anonfun$assemblyTask$1.apply(Assembly.scala:240) 
        at sbtassembly.Assembly$$anonfun$assemblyTask$1.apply(Assembly.scala:237) 
        at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47) 
        at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40) 
        at sbt.std.Transform$$anon$4.work(System.scala:63) 
        at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228) 
        at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228) 
        at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17) 
        at sbt.Execute.work(Execute.scala:237) 
        at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228) 
        at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228) 
        at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159) 
        at sbt.CompletionService$$anon$2.call(CompletionService.scala:28) 
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745) 
    java.lang.RuntimeException: Building python API received non-zero exit code 1 
        at scala.sys.package$.error(package.scala:27) 
        at PythonTasks$.buildPythonTask(PythonTasks.scala:25) 
        at $b9e935155022d705b7b0$$anonfun$jobServerPythonSettings$5.apply(build.sbt:110) 
        at $b9e935155022d705b7b0$$anonfun$jobServerPythonSettings$5.apply(build.sbt:110) 
        at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47) 
        at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40) 
        at sbt.std.Transform$$anon$4.work(System.scala:63) 
        at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228) 
        at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228) 
        at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17) 
        at sbt.Execute.work(Execute.scala:237) 
        at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228) 
        at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228) 
        at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159) 
        at sbt.CompletionService$$anon$2.call(CompletionService.scala:28) 
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745) 
    [error] (root/*:assembly) deduplicate: different file contents found in the following: 
    [error] /home/kmk/.ivy2/cache/io.netty/netty-all/jars/netty-all-4.0.37.Final.jar:META-INF/io.netty.versions.properties 
    [error] /home/kmk/.ivy2/cache/io.netty/netty-handler/jars/netty-handler-4.0.37.Final.jar:META-INF/io.netty.versions.properties 
    [error] /home/kmk/.ivy2/cache/io.netty/netty-buffer/jars/netty-buffer-4.0.37.Final.jar:META-INF/io.netty.versions.properties 
    [error] /home/kmk/.ivy2/cache/io.netty/netty-common/jars/netty-common-4.0.37.Final.jar:META-INF/io.netty.versions.properties 
    [error] /home/kmk/.ivy2/cache/io.netty/netty-transport/jars/netty-transport-4.0.37.Final.jar:META-INF/io.netty.versions.properties 
    [error] /home/kmk/.ivy2/cache/io.netty/netty-codec/jars/netty-codec-4.0.37.Final.jar:META-INF/io.netty.versions.properties 
    [error] (job-server-python/*:buildPython) Building python API received non-zero exit code 1 
    [error] Total time: 151 s, completed Apr 1, 2017 3:41:54 PM 
    

    *

    我在这里做错了什么?

    回答

    0

    根据你的错误信息,根据我的经验,我认为这个问题是由不同版本的冲突引起的,这些jar由相同的包名命名。所以一个正常的方法是尝试删除这些重复的罐子。或者你可以参考官方文档Excluding JARs and filessbt来解决通过合并策略的依赖关系,有一个SO线程Resolving Dependencies in creating JAR through SBT assembly这可能有助于你作为参考。

    同时,根据manual stepsspark-job-server部署,sbt job-server/assembly的操作似乎没有必要,这取决于您当前的环境。

    希望它有帮助。