使用org.apache.hadoop/*依赖项脱机编译sbt时遇到问题

8aqjt8rx  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(452)

sbt依赖org.apache.hadoop包,脱机编译时遇到很多问题。
一个简单的build.sbt:

name := "Test"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.hadoop" % "hadoop-yarn-api" % "2.2.0"

联机时工作正常,但脱机运行时出现以下错误,而包位于ivy缓存中(位于~/ivy2/cache/org.apache.hadoop/…):

[info] Loading project definition from /home/martin/Dev/S/project
[info] Set current project to Test (in build file:/home/martin/Dev/S/)
[info] Updating {file:/home/martin/Dev/S/}s...
[info] Resolving org.apache.hadoop#hadoop-yarn-api;2.2.0 ...
[warn] Host repo1.maven.org not found. url=https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-api/2.2.0/hadoop-yarn-api-2.2.0.pom
[info] You probably access the destination server through a proxy server that is not well configured.
[warn]  module not found: org.apache.hadoop#hadoop-yarn-api;2.2.0
[warn] ==== local: tried
[warn]   /home/martin/.ivy2/local/org.apache.hadoop/hadoop-yarn-api/2.2.0/ivys/ivy.xml
[warn] ==== public: tried
[warn]   https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-api/2.2.0/hadoop-yarn-api-2.2.0.pom
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: org.apache.hadoop#hadoop-yarn-api;2.2.0: not found
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn] 
[warn]  Note: Unresolved dependencies path:
[warn]      org.apache.hadoop:hadoop-yarn-api:2.2.0 (/home/martin/Dev/S/build.sbt#L15-16)
[warn]        +- test:test_2.10:1.0
sbt.ResolveException: unresolved dependency: org.apache.hadoop#hadoop-yarn-api;2.2.0: not found
    ...
[error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.hadoop#hadoop-yarn-api;2.2.0: not found
[error] Total time: 3 s, completed Apr 26, 2015 2:46:58 PM

添加以下解析程序没有帮助:

resolvers += Resolver.file("Local repo", file(System.getProperty("user.home") + "/.ivy2/cache")) (Resolver.ivyStylePatterns)

它只是增加了

[warn] ==== Local repo: tried
[warn]   /home/martin/.ivy2/cache/org.apache.hadoop/hadoop-yarn-api/2.2.0/ivys/ivy.xml

这些文件存在,但命名为ivy-2.2.0.xml,而不是2.2.0/ivys/ivy.xml
所以我试着加上

resolvers += Resolver.file("Local repo 2", file(System.getProperty("user.home") + "/.ivy2/cache")) ( Patterns("[organisation]/[module]/[artifact]-[revision].[ext]") )

强制命名约定,但它会在

[warn] ==== Local repo 2: tried
[warn]   /home/martin/.ivy2/cache/org/apache/hadoop/hadoop-yarn-api/ivy-2.2.0.xml

即使根据sbt文档,[组织]也应该是org.apache.hadoop而不是org/apache/hadoop
最后,作为最后的手段,我添加了一个丑陋的

resolvers += Resolver.file("Local hadoop cache", file(System.getProperty("user.home") + "/.ivy2/cache")) ( Patterns("org.apache.hadoop/[module]/[artifact]-[revision].[ext]") )

在那里它发现了一些东西,但仍然不满意:

[info] Loading project definition from /home/martin/Dev/S/project
[info] Set current project to Test (in build file:/home/martin/Dev/S/)
[info] Updating {file:/home/martin/Dev/S/}s...
[info] Resolving org.apache.hadoop#hadoop-yarn-api;2.2.0 ...
[warn] Host repo1.maven.org not found. url=https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-api/2.2.0/hadoop-yarn-api-2.2.0.pom
[info] You probably access the destination server through a proxy server that is not well configured.
[warn] xml parsing: ivy-2.2.0.xml.original:18:69: schema_reference.4: Failed to read schema document 'http://maven.apache.org/xsd/maven-4.0.0.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.
[warn] xml parsing: ivy-2.2.0.xml.original:19:11: schema_reference.4: Failed to read schema document 'http://maven.apache.org/xsd/maven-4.0.0.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.
[warn] xml parsing: ivy-2.2.0.xml.original:20:17: schema_reference.4: Failed to read schema document 'http://maven.apache.org/xsd/maven-4.0.0.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.
[warn] xml parsing: ivy-2.2.0.xml.original:21:14: schema_reference.4: Failed to read schema document 'http://maven.apache.org/xsd/maven-4.0.0.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.
[warn] xml parsing: ivy-2.2.0.xml.original:22:14: schema_reference.4: Failed to read schema document 'http://maven.apache.org/xsd/maven-4.0.0.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.
[warn] xml parsing: ivy-2.2.0.xml.original:24:17: schema_reference.4: Failed to read schema document 'http://maven.apache.org/xsd/maven-4.0.0.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.
[warn] xml parsing: ivy-2.2.0.xml.original:25:12: schema_reference.4: Failed to read schema document 'http://maven.apache.org/xsd/maven-4.0.0.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.
...

我没有线索知道下一步该做什么。如果我将.ivy2/cache/目录复制到脱机机器上,脱机构建对于我尝试的任何其他依赖项都可以正常工作。只是一堆org.apache.hadoop依赖关系导致了这个问题。.ivy2/cache/org.apache.hadoop下的结构和文件看起来与其他依赖项中的文件相同。
添加

offline := true

也没用。
使用sbt 0.13.7
谢谢!

pinkon5k

pinkon5k1#

似乎是一个sbt窃听器。我有同样的问题与sbt 0.13.8,但他们消失了与切换到0.13.9-rc3和清洁 ~/.ivy2

相关问题