Spring MVC和Apache Hadoop启动MapReduce作业

我试图通过Spring MVC应用程序启动hadoop MapReduce作业。Spring MVC和Apache Hadoop启动MapReduce作业

MVC应用程序使用Apache Tomcat 7.0.62正常工作。

hadoop应用程序也适用于spring-data-hadoop。

当我尝试合并这些项目时，MapReduce作业初始化与下面的错误一起崩溃。

15/05/31 16:10:18 WARN support.ClassPathXmlApplicationContext:  Exception encountered during context initialization - cancelling refresh attempt 
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'wordCountJob': Invocation of init method failed; nested exception is java.lang.UnsupportedOperationException: Not implemented by the DistributedFileSystem FileSystem implementation 
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1568) 
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:540) 
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476) 
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302) 
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:229) 
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298) 
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) 
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:706) 
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757) 
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480) 
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139) 
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83) 
at at.pixsearch.hadoop.Main.go(Main.java:19) 
at at.pixsearch.mvc.HelloController.hadoopTest(HelloController.java:22) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:606) 
at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215) 
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132) 
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) 
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:781) 
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:721) 
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:83) 
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:943) 
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:877) 
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966) 
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:857) 
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) 
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842) 
at javax.servlet.http.HttpServlet.service(HttpServlet.java:723) 
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) 
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) 
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) 
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) 
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) 
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) 
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) 
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) 
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861) 
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:620) 
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) 
at java.lang.Thread.run(Thread.java:745) 
Caused by: java.lang.UnsupportedOperationException: Not implemented by the DistributedFileSystem FileSystem implementation 
at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:216) 
at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2564) 
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2574) 
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591) 
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91) 
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630) 
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612) 
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370) 
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169) 
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:354) 
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) 
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:372) 
at org.springframework.data.hadoop.mapreduce.JobFactoryBean.afterPropertiesSet(JobFactoryBean.java:208) 
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1627) 
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1564)

控制器启动 “/ hadoop的” 映射

@Controller 
@RequestMapping("/") 
public class HelloController { 
    @RequestMapping(method = RequestMethod.GET) 
    public String printWelcome(ModelMap model) { 
     model.addAttribute("message", "Hello world!"); 
     return "hello"; 
    } 

    @RequestMapping(value = "/hadoop", method= RequestMethod.GET) 
    public void hadoopTest() { 
     Main.go(); 
    } 
}

类主要

public class Main { 
    static final Logger logger = LoggerFactory.getLogger(Main.class);  

    public static void go() { 
     ApplicationContext ctx = new ClassPathXmlApplicationContext("applicationContext.xml"); 

    }

}

的applicationContext.xml

<?xml version="1.0" encoding="UTF-8"?> 
<beans xmlns="http://www.springframework.org/schema/beans" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    xmlns:hdp="http://www.springframework.org/schema/hadoop" 
    xmlns:context="http://www.springframework.org/schema/context" 
    xsi:schemaLocation="http://www.springframework.org/schema/beans 
    http://www.springframework.org/schema/beans/spring-beans.xsd 
    http://www.springframework.org/schema/hadoop 
    http://www.springframework.org/schema/hadoop/spring-hadoop.xsd 
    http://www.springframework.org/schema/context 
    http://www.springframework.org/schema/context/spring-context.xsd"> 

    <context:property-placeholder location="classpath:application.properties" /> 

    <hdp:configuration> 
      fs.default.name=${hd.fs} 
      mapreduce.framework.name=${mr.fw} 
      xsrn.resourcemanager.address=${hd.rm} 
    </hdp:configuration> 

    <hdp:job id="wordCountJob" 
      input-path="${input.path}" 
      output-path="${output.path}" 
      jar-by-class="at.pixsearch.hadoop.Main" 
      mapper="at.pixsearch.hadoop.mapper.WordMapper" 
      reducer="at.pixsearch.hadoop.reducer.WordReducer"/> 

    <hdp:job-runner id="wordCountJobRunner" job-ref="wordCountJob" run-at-startup="true"/> 
</beans>

内作业

MVC-调度-servlet.xml中

<beans xmlns="http://www.springframework.org/schema/beans" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    xmlns:context="http://www.springframework.org/schema/context" 
    xmlns:mvc="http://www.springframework.org/schema/mvc" 
    xsi:schemaLocation="http://www.springframework.org/schema/beans 
    http://www.springframework.org/schema/beans/spring-beans.xsd 
    http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd http://www.springframework.org/schema/mvc http://www.springframework.org/schema/mvc/spring-mvc.xsd"> 

<context:component-scan base-package="at.pixsearch.mvc"/> 

<bean class="org.springframework.web.servlet.view.InternalResourceViewResolver"> 
    <property name="prefix" value="/WEB-INF/pages/"/> 
    <property name="suffix" value=".jsp"/> 
</bean> 

<mvc:annotation-driven/> 
<mvc:resources mapping="/resources/**" location="/resources/" /> 

</beans>

和我的pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> 

<modelVersion>4.0.0</modelVersion> 
<groupId>com.springapp</groupId> 
<artifactId>pixsearch</artifactId> 
<packaging>war</packaging> 
<version>1.0-SNAPSHOT</version> 
<name>pixsearch</name> 

<properties> 
    <spring.version>4.1.1.RELEASE</spring.version> 
</properties> 

<dependencies> 
    <dependency> 
     <groupId>org.springframework</groupId> 
     <artifactId>spring-core</artifactId> 
     <version>${spring.version}</version> 
    </dependency> 

    <dependency> 
     <groupId>org.springframework</groupId> 
     <artifactId>spring-web</artifactId> 
     <version>${spring.version}</version> 
    </dependency> 

    <dependency> 
     <groupId>javax.servlet</groupId> 
     <artifactId>servlet-api</artifactId> 
     <version>2.5</version> 
    </dependency> 

    <dependency> 
     <groupId>javax.servlet.jsp</groupId> 
     <artifactId>jsp-api</artifactId> 
     <version>2.1</version> 
     <scope>provided</scope> 
    </dependency> 

    <dependency> 
     <groupId>org.springframework</groupId> 
     <artifactId>spring-webmvc</artifactId> 
     <version>${spring.version}</version> 
    </dependency> 

    <dependency> 
     <groupId>org.springframework</groupId> 
     <artifactId>spring-test</artifactId> 
     <version>${spring.version}</version> 
     <scope>test</scope> 
    </dependency> 

    <dependency> 
     <groupId>junit</groupId> 
     <artifactId>junit</artifactId> 
     <version>4.11</version> 
     <scope>test</scope> 
    </dependency> 

    <dependency> 
     <groupId>jstl</groupId> 
     <artifactId>jstl</artifactId> 
     <version>1.2</version> 
    </dependency> 

    <dependency> 
     <groupId>org.springframework.data</groupId> 
     <artifactId>spring-data-hadoop</artifactId> 
     <version>2.1.2.RELEASE</version> 
    </dependency> 

    <dependency> 
     <groupId>org.apache.hadoop</groupId> 
     <artifactId>hadoop-core</artifactId> 
     <version>1.0.3</version> 
    </dependency> 
</dependencies> 

<build> 
    <finalName>pixsearch</finalName> 
    <plugins> 
     <plugin> 
      <artifactId>maven-compiler-plugin</artifactId> 
      <configuration> 
       <source>1.6</source> 
       <target>1.6</target> 
      </configuration> 
     </plugin> 
     <plugin> 
      <artifactId>maven-surefire-plugin</artifactId> 
      <configuration> 
       <includes> 
        <include>**/*Tests.java</include> 
       </includes> 
      </configuration> 
     </plugin> 
    </plugins> 
</build> 
</project>

项目结构如下：

-src 
    -main 
     -java 
      -at.pixsearch 
       -hadoop 
        -Mapper 
        -Reducer 
        Main.java 
       -mvc 
        -HelloController.java 
     -resources 
      applicationContext.xml 
      application.properties 
     -webapp 
      -resources 
       -js 
       -css 
       -public 
       -..... 
      -WEB-INF 
       -pages 
       -mvc-dispatcher-servlet.xml 
       -web.xml

林感谢任何帮助或提示：）

来源

2015-05-31 Michael Wutti

我通过配置hadoop解决了这个问题通过Java而不是.XML

所以

@RequestMapping(value="/hadoop" , method = RequestMethod.GET) 
public String hadoop(ModelMap model) throws URISyntaxException, IOException { 
    String result = null; 

    Configuration conf = new Configuration(); 
    FileSystem fs = FileSystem.get(new URI("hdfs://myhost:9000"), conf); 

    Path file = new Path("/user/michael/input/input1.txt"); 
    FSDataInputStream getIt = fs.open(file); 
    BufferedReader d = new BufferedReader(new InputStreamReader(getIt)); 

    String s = ""; 
    while ((s = d.readLine()) != null) { 
     result += s; 
    } 
    d.close(); 
    fs.close(); 

    model.addAttribute("result", result); 
    return "hello"; 
}

在控制器

能够访问HDFS。

来源

2015-05-31 17:50:46

Spring MVC和Apache Hadoop启动MapReduce作业

回答

相关问题