1
我有一个触发hadoop作业的spring批处理作业。请看下面我的工作配置。当我启动的工作,我得到下面的异常...spring批处理Hadoop失败,出现org.springframework.beans.factory.BeanNotOfRequiredTypeException
配置:
<hdp:job id="mr-my-job"
input-path="/data/input/"
output-path="/data/output/"
jar-by-class="org.test.Main"
mapper="org.test.Test1$Map"
combiner="org.test.Test1$Combiner"
reducer="org.test.Test1$ReduceFromCombiner"
number-reducers="7"
scope="step" />
例外:
org.springframework.beans.factory.BeanNotOfRequiredTypeException: Bean named 'mr-my-job' must be of type [org.apache.hadoop.mapreduce.Job], but was actually of type [$Proxy5]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:375)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.data.hadoop.mapreduce.JobExecutor.findJobs(JobExecutor.java:254)
at org.springframework.data.hadoop.mapreduce.JobExecutor.startJobs(JobExecutor.java:166)
at org.springframework.data.hadoop.batch.mapreduce.JobTasklet.execute(JobTasklet.java:90)
at org.springframework.data.hadoop.batch.mapreduce.JobTasklet$$FastClassBySpringCGLIB$$4805a065.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653)
at org.springframework.data.hadoop.batch.mapreduce.JobTasklet$$EnhancerBySpringCGLIB$$3be21557.execute(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at $Proxy2.execute(Unknown Source)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:406)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:330)
我加入CGLIB 3.1到类路径,并添加 <bean class="org.springframework.batch.core.scope.StepScope" p:proxyTargetClass="true" />
在我的XML,但仍然有相同的错误。
我调试,发现异常在个JobExecutor类抛出下面线而变换型[email protected]的豆到[email protected]。我认为CGLIB的自动代理在这里没有得到应用,因为它仍在使用JDK代理。
个JobExecutor类:
for (String name : jobNames) {
js.add(beanFactory.getBean(name, Job.class));
}
AbstractBeanFactory类
return getTypeConverter().convertIfNecessary(bean, requiredType);
请指点。
可能在某处(在Tasklet中,我想)你正在为一个具体的Hadoop作业类而不是一个接口进行布线。检查你的代码 –
感谢@LucaBassoRicci的回应。 我把我的工作配置放在原文中。你能检查我在哪里做错了吗? – user3222372
从“my-mr-job”中删除'scope =“step”' –