0
我在Yarn上运行我的Spark应用程序时遇到了一些问题。我有一个是没有任何问题，运行非常广泛的集成测试，但是当我在纱线运行该应用程序会引发以下错误：由于UTFDataFormatException导致的任务中不能序列化的任务：编码字符串太长
17/01/06 11:22:23 ERROR yarn.ApplicationMaster: User class threw exception: org.apache.spark.SparkException: Task not serializable
org.apache.spark.SparkException: Task not serializable
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:304)
at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:294)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:122)
at org.apache.spark.SparkContext.clean(SparkContext.scala:2067)
at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:324)
at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:323)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
at org.apache.spark.rdd.RDD.map(RDD.scala:323)
at org.apache.spark.sql.DataFrame.map(DataFrame.scala:1410)
at com.orgx.yy.dd.check.DQCheck$class.runDQCheck(DQCheck.scala:24)
at com.orgx.yy.dd.check.DQBatchCheck.runDQCheck(DQBatchCheck.scala:13)
at com.orgx.yy.dd.check.DQBatchCheck.doCheck(DQBatchCheck.scala:23)
at com.orgx.yy.dd.DQChecker$.main(DQChecker.scala:60)
at com.orgx.yy.dd.DQChecker.main(DQChecker.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542)
Caused by: java.io.UTFDataFormatException: encoded string too long: 72887 bytes
at java.io.DataOutputStream.writeUTF(DataOutputStream.java:364)
at java.io.DataOutputStream.writeUTF(DataOutputStream.java:323)
at com.typesafe.config.impl.SerializedConfigValue.writeValueData(SerializedConfigValue.java:295)
at com.typesafe.config.impl.SerializedConfigValue.writeValue(SerializedConfigValue.java:369)
at com.typesafe.config.impl.SerializedConfigValue.writeValueData(SerializedConfigValue.java:309)
at com.typesafe.config.impl.SerializedConfigValue.writeValue(SerializedConfigValue.java:369)
at com.typesafe.config.impl.SerializedConfigValue.writeValueData(SerializedConfigValue.java:309)
at com.typesafe.config.impl.SerializedConfigValue.writeValue(SerializedConfigValue.java:369)
at com.typesafe.config.impl.SerializedConfigValue.writeValueData(SerializedConfigValue.java:309)
at com.typesafe.config.impl.SerializedConfigValue.writeValue(SerializedConfigValue.java:369)
at com.typesafe.config.impl.SerializedConfigValue.writeExternal(SerializedConfigValue.java:435)
at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1459)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1430)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at scala.collection.immutable.$colon$colon.writeObject(List.scala:379)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:44)
at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:101)
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:301)
... 20 more
17/01/06 11:22:24 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.spark.SparkException: Task not serializable)
的罪魁祸首似乎是java.io.UTFDataFormatException：太长编码字符串：在72887字节。任何人都知道为什么会这样？
也许看看引发异常的源代码：'DataOutputStream.java：364' – maasg
@maasg我不确定这会是非常有用的，因为这是一些低级Java IO类，并且经常被使用许多图书馆。我怀疑该任务的序列化出了问题，并且正在触及某种大小限制。 – ErikHabanero
如果我要试着回答这样的问题，那就是我要开始的地方，只是想知道发生了什么问题。我同意这个问题可能在别的地方，但是你需要按照面包屑去bug巢穴。 – maasg