集成Dropwizard和Apache Spark

我正在使用Apache Spark实现在线分类服务。试图将应用程序提交给一个独立的集群这个剧本时，我有一个问题集成Dropwizard和Apache Spark

$SPARK_HOME/bin/spark-submit \ 
    --class com.example.msclassification.MscApplication\ 
    --master spark://192.168.55.165:7077 \ 
    --deploy-mode cluster \ 
    --executor-memory 2G \ 
    --total-executor-cores 4 \ 
    ./build/libs/msclassification-0.0.1-all.jar -server configuration.yml

它给我这样一个例外：

16/03/01 11:25:45 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
Exception in thread "main" java.lang.reflect.InvocationTargetException 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:606) 
    at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:58) 
    at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala) 
Caused by: java.lang.IllegalStateException: Unable to acquire the logger context 
    at io.dropwizard.logging.LoggingUtil.getLoggerContext(LoggingUtil.java:46) 
    at io.dropwizard.logging.BootstrapLogging.bootstrap(BootstrapLogging.java:45) 
    at io.dropwizard.logging.BootstrapLogging.bootstrap(BootstrapLogging.java:34) 
    at io.dropwizard.Application.<init>(Application.java:24) 
    at com.example.msclassification.MscApplication.<init>(MscApplication.groovy) 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) 
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526) 
    at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77) 
    at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102) 
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:57) 
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182) 
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:186) 
    at com.example.msclassification.MscApplication.main(MscApplication.groovy:23) 
    ... 6 more

测试时，出现了一个冲突记录和我已经解决了，使用以下gradle这个buildscript：

group 'com.example' 
version '0.0.1' 

apply plugin: 'groovy' 
apply plugin: 'com.github.johnrengelman.shadow' 
apply plugin: 'application' 

sourceCompatibility = 1.7 
mainClassName = "com.example.msclassification.MscApplication" 
repositories { 
    mavenCentral() 
} 
configurations.all { 
    resolutionStrategy { 
     force 'com.fasterxml.jackson.core:jackson-databind:2.4.4' 
    } 
} 
dependencies { 
    compile ('org.codehaus.groovy:groovy-all:2.3.11') 
    compile ("io.dropwizard:dropwizard-core:${project.properties.dropwizardVersion}") 
    compile ("io.dropwizard:dropwizard-jdbi:${project.properties.dropwizardVersion}") 
    compile ("org.elasticsearch:elasticsearch:${project.properties.elasticsearchVersion}") 
    compile 'org.slf4j:log4j-over-slf4j:1.7.18' 
    compile 'nz.ac.waikato.cms.weka:weka-dev:3.7.11' 
    compile 'org.codehaus.gpars:gpars:1.2.1' 
    compile ('commons-io:commons-io:2.4') 
    compile 'mysql:mysql-connector-java:5.1.38' 
    compile 'com.fasterxml.jackson.core:jackson-databind:2.4.4' 
    compile ('org.apache.spark:spark-core_2.10:1.6.0'){ 
     exclude group: 'org.slf4j' 
    } 
    compile ('org.apache.spark:spark-mllib_2.10:1.6.0'){ 
     exclude group: 'org.slf4j' 
    } 

// testCompile group: 'junit', name: 'junit', version: '4.11' 
    testCompile 'org.spockframework:spock-core:0.7-groovy-2.0' 
} 

buildscript { 
    repositories { jcenter() } 
    dependencies { 
     classpath 'com.github.jengelman.gradle.plugins:shadow:1.2.2' 
    } 
} 


//State the main entry and merge service files 
shadowJar{ 
    exclude 'META-INF/*.DSA' 
    exclude 'META-INF/*.RSA' 
    mergeServiceFiles() 
    zip64 true 
} 

runShadow{ 
    args 'server', "${project.properties.dropwizardConfig}" 
}

在这个剧本，我已经使用jackson databind 2.4.4和被迫210 revole软件包与Spark的冲突是使用老版本的jackson，它使用SLF4J LOG4J-12进行日志绑定，无论Dropwizard是否使用Logback，并且没有办法排除这种情况或强制它使用log4j而没有错误。你能提出一些建议吗？我会感激不尽的感谢！

来源

2016-03-01 innovatism

您的路径上可能包含log4j，从而导致getILoggerFactory返回this code中的LoggingContext以外的内容。

要找出哪些资源是log4j的拉动考虑运行：

gradle -q dependencyInsight --configuration compile --dependency log4j

星火实际上是在log4j的明确拉动的依赖关系（不只是通过SLF4J），这样很可能您的问题的根源。明确排除Spark依赖中的log4j，你应该很好。

来源

2016-03-01 21:05:41

集成Dropwizard和Apache Spark

回答

相关问题