sbt程序集由于文件冲突而失败

cgvd09ve  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(392)

我试图通过为我的项目运行sbt assembly来制作一个胖jar。我得到以下错误:

[error] (root/*:assembly) deduplicate: different file contents found in the following:
[error] /Users/xyz/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-core/jars/hadoop-mapreduce-client-core-2.2.0.jar:org/apache/hadoop/filecache/DistributedCache.class
[error] /Users/xyz/.ivy2/cache/org.apache.hadoop/hadoop-core/jars/hadoop-core-2.0.0-mr1-cdh4.7.1.jar:org/apache/hadoop/filecache/DistributedCache.class

hadoop mapreduce客户端核心的distributedcache现在不推荐使用。在my build.sbt中,我包括:

"org.apache.hadoop" % "hadoop-client" % "2.0.0-mr1-cdh4.7.1" excludeAll(
    ExclusionRule(organization = "javax.servlet"))

依赖关系如下:

org.apache.hadoop:hadoop-client:2.2.0 
 org.apache.hadoop:hadoop-mapreduce-client-app:2.2.0
  org.apache.hadoop:hadoop-mapreduce-client-core:2.2.0

我该怎么处理?
提前谢谢!

falq053o

falq053o1#

如果您打算删除mapreduce客户端应用程序的依赖项jar,那么在您依赖hadoop的情况下,该应用程序正在加载-client:2.2.0,只需添加不及物:

"org.apache.hadoop" % "hadoop-client" % "2.2.0" intransitive()

这将只包括hadoop-client:2.2.0 jar 并排除其所有依赖项。

相关问题