基本上我需要在电子病历上做一个棘手的工作。同样的工作在我的macbook上的本地hadoop上运行得非常好，但是在emr上的hadoop上失败了。
我也在努力在cascading user和scala用户组中获得关于这个问题的帮助，但是一直没有得到帮助。到目前为止，我还没有取得太大的进展后，尝试了各种变化，在过去的几天。
下面是我深入研究细节之前的错误：
线程“main”java.lang.nosuchmethoderror中出现异常：scala.predef$.arrowassoc（ljava/lang/object；）ljava/lang/object；在com.aggregation.job.dataaggregation$（dataaggregation。scala:30)位于com.aggregation.job.dataaggregation$。（dataaggregation.scala）位于com.aggregation.job.dataaggregation.main（dataaggregation.scala）位于sun.reflect.nativemethodaccessorimpl.invoke（nativemethodaccessorimpl）位于sun.reflect.nativemethodaccessorimpl.invoke（nativemethodaccessorimpl）的com.aggregation。java:57)在sun.reflect.delegatingmethodaccessorimpl.invoke（delegatingmethodaccessorimpl。java:43)在java.lang.reflect.method.invoke（方法。java:606)在org.apache.hadoop.util.runjar.main（runjar。java:212)
他们说这可能是scala与二进制文件的冲突，但我看不到任何明显的东西。如果有人能帮我弄清楚那就太好了。
环境：
amazonemrami:3.8.0（包括scala2.11.1、hadoop2.4.0、java1.7.0\u76-ami详细信息）
应用环境：滚烫0.15.0、scala 2.11.1、java 1.7.0\U 80、hadoop 2.4.0
我尝试过对配置进行各种更改，甚至在emr中手动安装了更高的scala版本，但到目前为止，这个错误并没有消失。
请帮帮我！谢谢您。
我
设置
内部版本.sbt：

scalaVersion := "2.11.1"

ivyScala := ivyScala.value map {
  _.copy(overrideScalaVersion = true)
}

依赖项.sbt：

import AssemblyKeys._

val hadoopVersion = "2.4.0"

val scaldingVersion = "0.15.0"

libraryDependencies ++= Seq(
  "com.twitter" %% "scalding-core" % scaldingVersion,
  "com.twitter" %% "scalding-json" % scaldingVersion,
  "com.twitter" %% "scalding-jdbc" % scaldingVersion,
  "com.github.nscala-time" %% "nscala-time" % "2.0.0",
  "org.apache.hadoop" % "hadoop-common" % hadoopVersion % "provided",
  "org.apache.hadoop" % "hadoop-mapreduce-client-core" % hadoopVersion % "provided"
)

excludedJars in assembly <<= (fullClasspath in assembly) map { cp =>
  val excludes = Set(
  "jsp-api-2.1-6.1.14.jar",
  "jsp-2.1-6.1.14.jar",
  "jasper-compiler-5.5.12.jar",
  "minlog-1.2.jar", // Otherwise causes conflicts with Kyro (which bundles it)
  "janino-2.5.16.jar", // Janino includes a broken signature, and is not needed anyway
  "commons-beanutils-core-1.8.0.jar", // Clash with each other and with commons-collections
  "commons-beanutils-1.7.0.jar", // "
  "hadoop-core-1.2.1.jar", // Provided by Amazon EMR. Delete this line if you're not on EMR
  "hadoop-tools-1.2.1.jar" // "
)
cp filter { jar => excludes(jar.data.getName) }
}

resolvers ++= Seq(
  "Conjars repo" at "http://conjars.org/repo"
)

装配.sbt

import AssemblyKeys._

assemblySettings

mergeStrategy in assembly := Merge.mergeStrategy

项目/build.properties：

sbt.version=0.13.1

项目/assembly.sbt：

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.0")

最后，这是使用sbt依赖关系图插件的依赖关系树，看看是否有正确的版本。抱歉，这是相当长的，如果我应该显示此信息任何其他方式，请建议。
sbt依赖树：

[info] com.abc.aggregator_2.11:0.1-20150628T184441 [S]
[info]   +-com.github.nscala-time:nscala-time_2.11:2.0.0 [S]
[info]   | +-joda-time:joda-time:2.7
[info]   | +-org.joda:joda-convert:1.2
[info]   | 
[info]   +-com.twitter:scalding-core_2.11:0.15.0 [S]
[info]   | +-cascading:cascading-core:2.6.1
[info]   | | +-org.codehaus.janino:janino:2.7.5
[info]   | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info]   | | | 
[info]   | | +-riffle:riffle:0.1-dev
[info]   | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info]   | | 
[info]   | +-cascading:cascading-hadoop:2.6.1
[info]   | | +-cascading:cascading-core:2.6.1
[info]   | |   +-org.codehaus.janino:janino:2.7.5
[info]   | |   | +-org.codehaus.janino:commons-compiler:2.7.5
[info]   | |   | 
[info]   | |   +-riffle:riffle:0.1-dev
[info]   | |   +-thirdparty:jgrapht-jdk1.6:0.8.1
[info]   | |   
[info]   | +-cascading:cascading-local:2.6.1
[info]   | | +-cascading:cascading-core:2.6.1
[info]   | | | +-org.codehaus.janino:janino:2.7.5
[info]   | | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info]   | | | | 
[info]   | | | +-riffle:riffle:0.1-dev
[info]   | | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info]   | | | 
[info]   | | +-com.google.guava:guava:14.0.1 (evicted by: 15.0)
[info]   | | +-com.google.guava:guava:15.0
[info]   | | +-org.slf4j:slf4j-api:1.6.6
[info]   | | +-org.slf4j:slf4j-api:1.7.2 (evicted by: 1.6.6)
[info]   | | 
[info]   | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info]   | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info]   | | 
[info]   | +-com.twitter:bijection-core_2.11:0.8.0 [S]
[info]   | +-com.twitter:chill-algebird_2.11:0.6.0 [S]
[info]   | | +-com.esotericsoftware.kryo:kryo:2.21
[info]   | | | +-com.esotericsoftware.minlog:minlog:1.2
[info]   | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info]   | | | | +-org.ow2.asm:asm:4.0
[info]   | | | | 
[info]   | | | +-org.objenesis:objenesis:1.2
[info]   | | | 
[info]   | | +-com.twitter:algebird-core_2.11:0.10.0 (evicted by: 0.10.1)
[info]   | | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info]   | | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info]   | | | 
[info]   | | +-com.twitter:chill_2.11:0.6.0 [S]
[info]   | |   +-com.esotericsoftware.kryo:kryo:2.21
[info]   | |   | +-com.esotericsoftware.minlog:minlog:1.2
[info]   | |   | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info]   | |   | | +-org.ow2.asm:asm:4.0
[info]   | |   | | 
[info]   | |   | +-org.objenesis:objenesis:1.2
[info]   | |   | 
[info]   | |   +-com.twitter:chill-java:0.6.0
[info]   | |     +-com.esotericsoftware.kryo:kryo:2.21
[info]   | |       +-com.esotericsoftware.minlog:minlog:1.2
[info]   | |       +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info]   | |       | +-org.ow2.asm:asm:4.0
[info]   | |       | 
[info]   | |       +-org.objenesis:objenesis:1.2
[info]   | |       
[info]   | +-com.twitter:chill-hadoop:0.6.0
[info]   | | +-com.esotericsoftware.kryo:kryo:2.21
[info]   | | | +-com.esotericsoftware.minlog:minlog:1.2
[info]   | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info]   | | | | +-org.ow2.asm:asm:4.0
[info]   | | | | 
[info]   | | | +-org.objenesis:objenesis:1.2
[info]   | | | 
[info]   | | +-com.twitter:chill-java:0.6.0
[info]   | | | +-com.esotericsoftware.kryo:kryo:2.21
[info]   | | |   +-com.esotericsoftware.minlog:minlog:1.2
[info]   | | |   +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info]   | | |   | +-org.ow2.asm:asm:4.0
[info]   | | |   | 
[info]   | | |   +-org.objenesis:objenesis:1.2
[info]   | | |   
[info]   | | +-org.slf4j:slf4j-api:1.6.6
[info]   | | 
[info]   | +-com.twitter:chill-java:0.6.0
[info]   | | +-com.esotericsoftware.kryo:kryo:2.21
[info]   | |   +-com.esotericsoftware.minlog:minlog:1.2
[info]   | |   +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info]   | |   | +-org.ow2.asm:asm:4.0
[info]   | |   | 
[info]   | |   +-org.objenesis:objenesis:1.2
[info]   | |   
[info]   | +-com.twitter:chill_2.11:0.6.0 [S]
[info]   | | +-com.esotericsoftware.kryo:kryo:2.21
[info]   | | | +-com.esotericsoftware.minlog:minlog:1.2
[info]   | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info]   | | | | +-org.ow2.asm:asm:4.0
[info]   | | | | 
[info]   | | | +-org.objenesis:objenesis:1.2
[info]   | | | 
[info]   | | +-com.twitter:chill-java:0.6.0
[info]   | |   +-com.esotericsoftware.kryo:kryo:2.21
[info]   | |     +-com.esotericsoftware.minlog:minlog:1.2
[info]   | |     +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info]   | |     | +-org.ow2.asm:asm:4.0
[info]   | |     | 
[info]   | |     +-org.objenesis:objenesis:1.2
[info]   | |     
[info]   | +-com.twitter:maple:0.15.0
[info]   | | +-cascading:cascading-hadoop:2.6.1
[info]   | |   +-cascading:cascading-core:2.6.1
[info]   | |     +-org.codehaus.janino:janino:2.7.5
[info]   | |     | +-org.codehaus.janino:commons-compiler:2.7.5
[info]   | |     | 
[info]   | |     +-riffle:riffle:0.1-dev
[info]   | |     +-thirdparty:jgrapht-jdk1.6:0.8.1
[info]   | |     
[info]   | +-com.twitter:scalding-args_2.11:0.15.0 [S]
[info]   | +-com.twitter:scalding-date_2.11:0.15.0 [S]
[info]   | +-com.twitter:scalding-serialization_2.11:0.15.0 [S]
[info]   | +-org.slf4j:slf4j-api:1.6.6
[info]   | 
[info]   +-com.twitter:scalding-jdbc_2.11:0.15.0 [S]
[info]   | +-cascading:cascading-jdbc-core:2.6.0
[info]   | | +-com.google.guava:guava:15.0
[info]   | | 
[info]   | +-cascading:cascading-jdbc-mysql:2.6.0
[info]   | | +-cascading:cascading-jdbc-core:2.6.0
[info]   | | | +-com.google.guava:guava:15.0
[info]   | | | 
[info]   | | +-com.google.guava:guava:15.0
[info]   | | +-mysql:mysql-connector-java:5.1.25
[info]   | | 
[info]   | +-com.twitter:scalding-core_2.11:0.15.0 [S]
[info]   |   +-cascading:cascading-core:2.6.1
[info]   |   | +-org.codehaus.janino:janino:2.7.5
[info]   |   | | +-org.codehaus.janino:commons-compiler:2.7.5
[info]   |   | | 
[info]   |   | +-riffle:riffle:0.1-dev
[info]   |   | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info]   |   | 
[info]   |   +-cascading:cascading-hadoop:2.6.1
[info]   |   | +-cascading:cascading-core:2.6.1
[info]   |   |   +-org.codehaus.janino:janino:2.7.5
[info]   |   |   | +-org.codehaus.janino:commons-compiler:2.7.5
[info]   |   |   | 
[info]   |   |   +-riffle:riffle:0.1-dev
[info]   |   |   +-thirdparty:jgrapht-jdk1.6:0.8.1
[info]   |   |   
[info]   |   +-cascading:cascading-local:2.6.1
[info]   |   | +-cascading:cascading-core:2.6.1
[info]   |   | | +-org.codehaus.janino:janino:2.7.5
[info]   |   | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info]   |   | | | 
[info]   |   | | +-riffle:riffle:0.1-dev
[info]   |   | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info]   |   | | 
[info]   |   | +-com.google.guava:guava:14.0.1 (evicted by: 15.0)
[info]   |   | +-com.google.guava:guava:15.0
[info]   |   | +-org.slf4j:slf4j-api:1.6.6
[info]   |   | +-org.slf4j:slf4j-api:1.7.2 (evicted by: 1.6.6)
[info]   |   | 
[info]   |   +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info]   |   | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info]   |   | 
[info]   |   +-com.twitter:bijection-core_2.11:0.8.0 [S]
[info]   |   +-com.twitter:chill-algebird_2.11:0.6.0 [S]
[info]   |   | +-com.esotericsoftware.kryo:kryo:2.21
[info]   |   | | +-com.esotericsoftware.minlog:minlog:1.2
[info]   |   | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info]   |   | | | +-org.ow2.asm:asm:4.0
[info]   |   | | | 
[info]   |   | | +-org.objenesis:objenesis:1.2
[info]   |   | | 
[info]   |   | +-com.twitter:algebird-core_2.11:0.10.0 (evicted by: 0.10.1)
[info]   |   | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info]   |   | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info]   |   | | 
[info]   |   | +-com.twitter:chill_2.11:0.6.0 [S]
[info]   |   |   +-com.esotericsoftware.kryo:kryo:2.21
[info]   |   |   | +-com.esotericsoftware.minlog:minlog:1.2
[info]   |   |   | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info]   |   |   | | +-org.ow2.asm:asm:4.0
[info]   |   |   | | 
[info]   |   |   | +-org.objenesis:objenesis:1.2
[info]   |   |   | 
[info]   |   |   +-com.twitter:chill-java:0.6.0
[info]   |   |     +-com.esotericsoftware.kryo:kryo:2.21
[info]   |   |       +-com.esotericsoftware.minlog:minlog:1.2
[info]   |   |       +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info]   |   |       | +-org.ow2.asm:asm:4.0
[info]   |   |       | 
[info]   |   |       +-org.objenesis:objenesis:1.2
[info]   |   |       
[info]   |   +-com.twitter:chill-hadoop:0.6.0
[info]   |   | +-com.esotericsoftware.kryo:kryo:2.21
[info]   |   | | +-com.esotericsoftware.minlog:minlog:1.2
[info]   |   | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info]   |   | | | +-org.ow2.asm:asm:4.0
[info]   |   | | | 
[info]   |   | | +-org.objenesis:objenesis:1.2
[info]   |   | | 
[info]   |   | +-com.twitter:chill-java:0.6.0
[info]   |   | | +-com.esotericsoftware.kryo:kryo:2.21
[info]   |   | |   +-com.esotericsoftware.minlog:minlog:1.2
[info]   |   | |   +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info]   |   | |   | +-org.ow2.asm:asm:4.0
[info]   |   | |   | 
[info]   |   | |   +-org.objenesis:objenesis:1.2
[info]   |   | |   
[info]   |   | +-org.slf4j:slf4j-api:1.6.6
[info]   |   | 
[info]   |   +-com.twitter:chill-java:0.6.0
[info]   |   | +-com.esotericsoftware.kryo:kryo:2.21
[info]   |   |   +-com.esotericsoftware.minlog:minlog:1.2
[info]   |   |   +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info]   |   |   | +-org.ow2.asm:asm:4.0
[info]   |   |   | 
[info]   |   |   +-org.objenesis:objenesis:1.2
[info]   |   |   
[info]   |   +-com.twitter:chill_2.11:0.6.0 [S]
[info]   |   | +-com.esotericsoftware.kryo:kryo:2.21
[info]   |   | | +-com.esotericsoftware.minlog:minlog:1.2
[info]   |   | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info]   |   | | | +-org.ow2.asm:asm:4.0
[info]   |   | | | 
[info]   |   | | +-org.objenesis:objenesis:1.2
[info]   |   | | 
[info]   |   | +-com.twitter:chill-java:0.6.0
[info]   |   |   +-com.esotericsoftware.kryo:kryo:2.21
[info]   |   |     +-com.esotericsoftware.minlog:minlog:1.2
[info]   |   |     +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info]   |   |     | +-org.ow2.asm:asm:4.0
[info]   |   |     | 
[info]   |   |     +-org.objenesis:objenesis:1.2
[info]   |   |     
[info]   |   +-com.twitter:maple:0.15.0
[info]   |   | +-cascading:cascading-hadoop:2.6.1
[info]   |   |   +-cascading:cascading-core:2.6.1
[info]   |   |     +-org.codehaus.janino:janino:2.7.5
[info]   |   |     | +-org.codehaus.janino:commons-compiler:2.7.5
[info]   |   |     | 
[info]   |   |     +-riffle:riffle:0.1-dev
[info]   |   |     +-thirdparty:jgrapht-jdk1.6:0.8.1
[info]   |   |     
[info]   |   +-com.twitter:scalding-args_2.11:0.15.0 [S]
[info]   |   +-com.twitter:scalding-date_2.11:0.15.0 [S]
[info]   |   +-com.twitter:scalding-serialization_2.11:0.15.0 [S]
[info]   |   +-org.slf4j:slf4j-api:1.6.6
[info]   |   
[info]   +-com.twitter:scalding-json_2.11:0.15.0 [S]
[info]     +-com.fasterxml.jackson.module:jackson-module-scala_2.11:2.4.2 [S]
[info]     | +-com.fasterxml.jackson.core:jackson-annotations:2.4.2
[info]     | +-com.fasterxml.jackson.core:jackson-core:2.4.2
[info]     | +-com.fasterxml.jackson.core:jackson-databind:2.4.2
[info]     | | +-com.fasterxml.jackson.core:jackson-annotations:2.4.0 (evicted by: 2.4.2)
[info]     | | +-com.fasterxml.jackson.core:jackson-annotations:2.4.2
[info]     | | +-com.fasterxml.jackson.core:jackson-core:2.4.2
[info]     | | 
[info]     | +-com.google.code.findbugs:jsr305:2.0.1
[info]     | +-com.google.guava:guava:15.0
[info]     | +-com.thoughtworks.paranamer:paranamer:2.6
[info]     | +-org.scala-lang:scala-reflect:2.11.2 [S]
[info]     | 
[info]     +-com.twitter:scalding-core_2.11:0.15.0 [S]
[info]     | +-cascading:cascading-core:2.6.1
[info]     | | +-org.codehaus.janino:janino:2.7.5
[info]     | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info]     | | | 
[info]     | | +-riffle:riffle:0.1-dev
[info]     | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info]     | | 
[info]     | +-cascading:cascading-hadoop:2.6.1
[info]     | | +-cascading:cascading-core:2.6.1
[info]     | |   +-org.codehaus.janino:janino:2.7.5
[info]     | |   | +-org.codehaus.janino:commons-compiler:2.7.5
[info]     | |   | 
[info]     | |   +-riffle:riffle:0.1-dev
[info]     | |   +-thirdparty:jgrapht-jdk1.6:0.8.1
[info]     | |   
[info]     | +-cascading:cascading-local:2.6.1
[info]     | | +-cascading:cascading-core:2.6.1
[info]     | | | +-org.codehaus.janino:janino:2.7.5
[info]     | | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info]     | | | | 
[info]     | | | +-riffle:riffle:0.1-dev
[info]     | | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info]     | | | 
[info]     | | +-com.google.guava:guava:14.0.1 (evicted by: 15.0)
[info]     | | +-com.google.guava:guava:15.0
[info]     | | +-org.slf4j:slf4j-api:1.6.6
[info]     | | +-org.slf4j:slf4j-api:1.7.2 (evicted by: 1.6.6)
[info]     | | 
[info]     | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info]     | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info]     | | 
[info]     | +-com.twitter:bijection-core_2.11:0.8.0 [S]
[info]     | +-com.twitter:chill-algebird_2.11:0.6.0 [S]
[info]     | | +-com.esotericsoftware.kryo:kryo:2.21
[info]     | | | +-com.esotericsoftware.minlog:minlog:1.2
[info]     | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info]     | | | | +-org.ow2.asm:asm:4.0
[info]     | | | | 
[info]     | | | +-org.objenesis:objenesis:1.2
[info]     | | | 
[info]     | | +-com.twitter:algebird-core_2.11:0.10.0 (evicted by: 0.10.1)
[info]     | | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info]     | | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info]     | | | 
[info]     | | +-com.twitter:chill_2.11:0.6.0 [S]
[info]     | |   +-com.esotericsoftware.kryo:kryo:2.21
[info]     | |   | +-com.esotericsoftware.minlog:minlog:1.2
[info]     | |   | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info]     | |   | | +-org.ow2.asm:asm:4.0
[info]     | |   | | 
[info]     | |   | +-org.objenesis:objenesis:1.2
[info]     | |   | 
[info]     | |   +-com.twitter:chill-java:0.6.0
[info]     | |     +-com.esotericsoftware.kryo:kryo:2.21
[info]     | |       +-com.esotericsoftware.minlog:minlog:1.2
[info]     | |       +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info]     | |       | +-org.ow2.asm:asm:4.0
[info]     | |       | 
[info]     | |       +-org.objenesis:objenesis:1.2
[info]     | |       
[info]     | +-com.twitter:chill-hadoop:0.6.0
[info]     | | +-com.esotericsoftware.kryo:kryo:2.21
[info]     | | | +-com.esotericsoftware.minlog:minlog:1.2
[info]     | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info]     | | | | +-org.ow2.asm:asm:4.0
[info]     | | | | 
[info]     | | | +-org.objenesis:objenesis:1.2
[info]     | | | 
[info]     | | +-com.twitter:chill-java:0.6.0
[info]     | | | +-com.esotericsoftware.kryo:kryo:2.21
[info]     | | |   +-com.esotericsoftware.minlog:minlog:1.2
[info]     | | |   +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info]     | | |   | +-org.ow2.asm:asm:4.0
[info]     | | |   | 
[info]     | | |   +-org.objenesis:objenesis:1.2
[info]     | | |   
[info]     | | +-org.slf4j:slf4j-api:1.6.6
[info]     | | 
[info]     | +-com.twitter:chill-java:0.6.0
[info]     | | +-com.esotericsoftware.kryo:kryo:2.21
[info]     | |   +-com.esotericsoftware.minlog:minlog:1.2
[info]     | |   +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info]     | |   | +-org.ow2.asm:asm:4.0
[info]     | |   | 
[info]     | |   +-org.objenesis:objenesis:1.2
[info]     | |   
[info]     | +-com.twitter:chill_2.11:0.6.0 [S]
[info]     | | +-com.esotericsoftware.kryo:kryo:2.21
[info]     | | | +-com.esotericsoftware.minlog:minlog:1.2
[info]     | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info]     | | | | +-org.ow2.asm:asm:4.0
[info]     | | | | 
[info]     | | | +-org.objenesis:objenesis:1.2
[info]     | | | 
[info]     | | +-com.twitter:chill-java:0.6.0
[info]     | |   +-com.esotericsoftware.kryo:kryo:2.21
[info]     | |     +-com.esotericsoftware.minlog:minlog:1.2
[info]     | |     +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info]     | |     | +-org.ow2.asm:asm:4.0
[info]     | |     | 
[info]     | |     +-org.objenesis:objenesis:1.2
[info]     | |     
[info]     | +-com.twitter:maple:0.15.0
[info]     | | +-cascading:cascading-hadoop:2.6.1
[info]     | |   +-cascading:cascading-core:2.6.1
[info]     | |     +-org.codehaus.janino:janino:2.7.5
[info]     | |     | +-org.codehaus.janino:commons-compiler:2.7.5
[info]     | |     | 
[info]     | |     +-riffle:riffle:0.1-dev
[info]     | |     +-thirdparty:jgrapht-jdk1.6:0.8.1
[info]     | |     
[info]     | +-com.twitter:scalding-args_2.11:0.15.0 [S]
[info]     | +-com.twitter:scalding-date_2.11:0.15.0 [S]
[info]     | +-com.twitter:scalding-serialization_2.11:0.15.0 [S]
[info]     | +-org.slf4j:slf4j-api:1.6.6
[info]     | 
[info]     +-org.json4s:json4s-native_2.11:3.2.11 [S]
[info]       +-org.json4s:json4s-core_2.11:3.2.11 [S]
[info]         +-com.thoughtworks.paranamer:paranamer:2.6
[info]         +-org.json4s:json4s-ast_2.11:3.2.11 [S]
[info]         +-org.scala-lang:scalap:2.11.0
[info]           +-org.scala-lang:scala-compiler:2.11.1 [S]
[info]             +-org.scala-lang.modules:scala-parser-combinators_2.11:1.0.1 [S]
[info]             +-org.scala-lang.modules:scala-xml_2.11:1.0.2 [S]
[info]             +-org.scala-lang:scala-reflect:2.11.1 (evicted by: 2.11.2)
[info]             +-org.scala-lang:scala-reflect:2.11.2 [S]
[info]             
[success] Total time: 12 s, completed Jun 28, 2015 11:44:54 AM

根据要求提供更多信息：
我使用“sbt assembly”构建fat jar，目前我正在使用aws控制台和一个“customjar”步骤来测试这个过程，然后再自动化这个过程。

JAR location: s3://path/to/jar/data-aggregator-0.1.jar
Arguments: com.abc.aggregation.job.DataAggregation --hdfs --input s3n://path/to/input/data/file.json --output s3n://path/to/input/data/file.txt

更新：
通过提供指向scala2.11.1jars的hadoop\u类路径，我克服了上述错误，同时将其从sbt组装步骤中排除。这是使用hadoop-user-env.sh传入的，似乎适用于主节点。然而，一旦它到达mapper步骤，它就再次失败，出现了另一个scala错误。现在我被困在这一步。
假设这是因为Map器和还原器没有看到hadoop\u类路径的更新，我尝试包含-libjars参数，该参数指向hadoop master本身上的scala jar文件。但这个（下图）似乎不起作用。

JAR location: s3://path/to/jar/data-aggregator-0.1.jar
Arguments: com.abc.aggregation.job.DataAggregation -libjars /usr/share/scala/lib/scala-library.jar,/usr/share/scala/lib/scala-reflect.jar --hdfs --input s3n://path/to/input/data/file.json --output s3n://path/to/input/data/file.txt

2条答案

按热度按时间

5jdjgkvh1#

在与类似问题斗争之后。。。。。这显然是jar依赖项之间的版本不匹配。我首先登录到emr主程序，然后打开sparkshell显示正在使用的scala版本。现在在pom文件中配置您自己的依赖项，它应该可以工作了。

赞(0）回复(0）举报 2021-05-30

2guxujil2#

固定的。所以在emr示例中确实有多个scala jar，它们不是来自我的应用程序jar。
2.10jar隐藏在/usr/share/aws/emr/emrfs/lib中，与/usr/share/scala下2.11二进制文件的安装位置不同。所以我在集群的所有示例中都去掉了2.10jar，并且我的工作成功地完成了。现在我将为此创建一个引导操作。

$ sudo find / -name "scala-library-2.10.*.jar" -exec rm -rf {} \;

仅供参考，以下是它的路径：

[ec2-user@ip-172-31-72-130 ~]$ sudo find / -name "scala-library-2.11.*.jar"

/home/hadoop/.versions/hbase-0.94.18/lib/scala-library-2.11.0.jar
/usr/share/doc/scala/api/jars/scala-library-2.11.1-javadoc.jar

[ec2-user@ip-172-31-72-130 ~]$ sudo find / -name "scala-library-2.10.*.jar"

/usr/share/aws/emr/emrfs/lib/scala-library-2.10.5.jar

emr上的烫伤：hadoop作业失败，错误为nosuchmethoderror:scala.predef$.arrowassoc(ljava/lang/object；)ljava/lang/object；

2条答案

相关问题

热门标签

最新问答