用hadoop构建giraph

8tntrjer  于 2021-06-03  发布在  Hadoop
关注(0)|答案(2)|浏览(486)

我正在尝试用hadoop2.7.1设置giraph，尽管我可能会尝试，但它似乎不起作用。
我尝试了以下链接，
http://giraph.apache.org/build.html
https://blog.cloudera.com/blog/2014/02/how-to-write-and-run-giraph-jobs-on-hadoop/
http://lab.hypotheses.org/1207
当我尝试使用 mvn -Phadoop_yarn -Dhadoop.version=2.7.1 -DskipTests package ```
hadoopuser:/usr/local/giraph$ mvn -Phadoop_yarn -Dhadoop.version=2.7.1 -DskipTests package
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Apache Giraph Parent
[INFO] Apache Giraph Core
[INFO] Apache Giraph Examples
[INFO] Apache Giraph Distribution
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Giraph Parent 1.1.0
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- mavanagaiata:0.5.0:branch (git-commit) @ giraph-parent ---
[INFO]
[INFO] --- mavanagaiata:0.5.0:commit (git-commit) @ giraph-parent ---
[INFO]
[INFO] --- maven-enforcer-plugin:1.2:enforce (default) @ giraph-parent ---
[INFO] Skipping Rule Enforcement.
[INFO]
[INFO] --- maven-dependency-versions-check-plugin:2.0.2:check (default) @ giraph-parent ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.4:process (default) @ giraph-parent ---
[INFO]
[INFO] --- maven-dependency-plugin:2.8:analyze-only (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO]
[INFO] --- maven-dependency-plugin:2.8:analyze-duplicate (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO]
[INFO] --- maven-dependency-plugin:2.8:analyze-dep-mgt (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO]
[INFO] --- maven-duplicate-finder-plugin:1.0.4:check (default) @ giraph-parent ---
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ giraph-parent ---
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Giraph Core 1.1.0
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- mavanagaiata:0.5.0:branch (git-commit) @ giraph-core ---
[INFO]
[INFO] --- mavanagaiata:0.5.0:commit (git-commit) @ giraph-core ---
[INFO]
[INFO] --- maven-enforcer-plugin:1.2:enforce (default) @ giraph-core ---
[INFO] Skipping Rule Enforcement.
[INFO]
[INFO] --- maven-dependency-versions-check-plugin:2.0.2:check (default) @ giraph-core ---
[INFO]
[INFO] --- munge-maven-plugin:1.0:munge (munge) @ giraph-core ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.4:process (default) @ giraph-core ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ giraph-core ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ giraph-core ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 711 source files to /usr/local/giraph/giraph-core/target/munged/classes
[INFO] -------------------------------------------------------------
[WARNING] COMPILATION WARNING :
[INFO] -------------------------------------------------------------
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[42,29] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[44,25] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[57,32] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[39,32] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[42,29] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[44,25] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/DistributedCacheUtils.java: Some input files use or override a deprecated API.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/DistributedCacheUtils.java: Recompile with -Xlint:deprecation for details.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/TestGraph.java: Some input files use unchecked or unsafe operations.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/TestGraph.java: Recompile with -Xlint:unchecked for details.
[INFO] 10 warnings
[INFO] -------------------------------------------------------------
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68] cannot find symbol
symbol: variable SASL_PROPS
location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62] cannot find symbol
symbol: variable SASL_PROPS
location: class org.apache.hadoop.security.SaslRpcServer
[INFO] 2 errors
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Giraph Parent ............................... SUCCESS [ 4.374 s]
[INFO] Apache Giraph Core ................................. FAILURE [ 8.469 s]
[INFO] Apache Giraph Examples ............................. SKIPPED
[INFO] Apache Giraph Distribution ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13.136 s
[INFO] Finished at: 2015-11-16T10:19:07+05:30
[INFO] Final Memory: 45M/335M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project giraph-core: Compilation failure: Compilation failure:
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68] cannot find symbol
[ERROR] symbol: variable SASL_PROPS
[ERROR] location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62] cannot find symbol
[ERROR] symbol: variable SASL_PROPS
[ERROR] location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :giraph-cor

当我试着用， `mvn -Phadoop_yarn -Dhadoop.version=2.7.1 -DskipTests install` ```
hadoopuser:/usr/local/giraph$ mvn -Phadoop_yarn -Dhadoop.version=2.7.1 -DskipTests install
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] Apache Giraph Parent
[INFO] Apache Giraph Core
[INFO] Apache Giraph Examples
[INFO] Apache Giraph Distribution
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Giraph Parent 1.1.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- mavanagaiata:0.5.0:branch (git-commit) @ giraph-parent ---
[INFO] 
[INFO] --- mavanagaiata:0.5.0:commit (git-commit) @ giraph-parent ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.2:enforce (default) @ giraph-parent ---
[INFO] Skipping Rule Enforcement.
[INFO] 
[INFO] --- maven-dependency-versions-check-plugin:2.0.2:check (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.4:process (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-only (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-duplicate (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-dep-mgt (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-duplicate-finder-plugin:1.0.4:check (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ giraph-parent ---
[INFO] 
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ giraph-parent ---
[INFO] Installing /usr/local/giraph/pom.xml to /home/hadoopuser/.m2/repository/org/apache/giraph/giraph-parent/1.1.0/giraph-parent-1.1.0.pom
[INFO] Installing /usr/local/giraph/target/giraph-parent-1.1.0-site.xml to /home/hadoopuser/.m2/repository/org/apache/giraph/giraph-parent/1.1.0/giraph-parent-1.1.0-site.xml
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Giraph Core 1.1.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- mavanagaiata:0.5.0:branch (git-commit) @ giraph-core ---
[INFO] 
[INFO] --- mavanagaiata:0.5.0:commit (git-commit) @ giraph-core ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.2:enforce (default) @ giraph-core ---
[INFO] Skipping Rule Enforcement.
[INFO] 
[INFO] --- maven-dependency-versions-check-plugin:2.0.2:check (default) @ giraph-core ---
[INFO] 
[INFO] --- munge-maven-plugin:1.0:munge (munge) @ giraph-core ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.4:process (default) @ giraph-core ---
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ giraph-core ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ giraph-core ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 711 source files to /usr/local/giraph/giraph-core/target/munged/classes
[INFO] -------------------------------------------------------------
[WARNING] COMPILATION WARNING : 
[INFO] -------------------------------------------------------------
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[42,29] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[44,25] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[57,32] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[39,32] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[42,29] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[44,25] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/DistributedCacheUtils.java: Some input files use or override a deprecated API.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/DistributedCacheUtils.java: Recompile with -Xlint:deprecation for details.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/TestGraph.java: Some input files use unchecked or unsafe operations.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/TestGraph.java: Recompile with -Xlint:unchecked for details.
[INFO] 10 warnings 
[INFO] -------------------------------------------------------------
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68] cannot find symbol
  symbol:   variable SASL_PROPS
  location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62] cannot find symbol
  symbol:   variable SASL_PROPS
  location: class org.apache.hadoop.security.SaslRpcServer
[INFO] 2 errors 
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Giraph Parent ............................... SUCCESS [  4.360 s]
[INFO] Apache Giraph Core ................................. FAILURE [  7.473 s]
[INFO] Apache Giraph Examples ............................. SKIPPED
[INFO] Apache Giraph Distribution ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12.122 s
[INFO] Finished at: 2015-11-16T10:26:04+05:30
[INFO] Final Memory: 46M/374M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project giraph-core: Compilation failure: Compilation failure:
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68] cannot find symbol
[ERROR] symbol:   variable SASL_PROPS
[ERROR] location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62] cannot find symbol
[ERROR] symbol:   variable SASL_PROPS
[ERROR] location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :giraph-core

---
Version information follows,

Hadoop:  

    hadoopuser:~$ hadoop version
    Hadoop 2.7.1
    Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 15ecc87ccf4a0228f35af08fc56de536e6ce657a
    Compiled by jenkins on 2015-06-29T06:04Z
    Compiled with protoc 2.5.0
    From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58a
    This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar

Giraph is version 1.1.0 downloaded from here : http://archive.apache.org/dist/giraph/giraph-1.1.0/giraph-dist-1.1.0-src.tar.bz2

Java

    hadoopuser:~$ java -version
    java version "1.8.0_05"
    Java(TM) SE Runtime Environment (build 1.8.0_05-b13)
    Java HotSpot(TM) 64-Bit Server VM (build 25.5-b02, mixed mode)

Maven 

    hadoopuser:~$ mvn -version
    Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T17:27:37+05:30)
    Maven home: /opt/apache-maven-3.3.3
    Java version: 1.8.0_05, vendor: Oracle Corporation
    Java home: /usr/lib/jvm/jdk1.8.0/jre
    Default locale: en_IN, platform encoding: UTF-8
    OS name: "linux", version: "3.13.0-68-generic", arch: "amd64", family: "unix"

编辑：

正如史蒂夫·凯瑟在他的回答中建议的那样，我修改了pom.xlm，修改的部分如下所示。

<properties>
<hadoop.version>SET_HADOOP_VERSION_USING_MVN_DASH_D_OPTION</hadoop.version>
    <munge.symbols>PURE_YARN</munge.symbols>
    <!-- TODO: add these checks eventually -->
    <project.enforcer.skip>true</project.enforcer.skip>
    <giraph.maven.dependency.plugin.skip>true</giraph.maven.dependency.plugin.skip>
    <giraph.maven.duplicate.finder.skip>true</giraph.maven.duplicate.finder.skip>
  </properties>

在我跑步的时候做了这些 mvn -Phadoop_yarn -Dhadoop.version=2.7.1 -DskipTests install 我得到了大量的输出，但失败了。部分输出如下所示，

[INFO] 
[INFO] --- apache-rat-plugin:0.8:check (default) @ giraph-core ---
[INFO] Exclude: CODE_CONVENTIONS
[INFO] Exclude:**/target/**
[INFO] Exclude: /_bsp/**
[INFO] Exclude: .checkstyle
[INFO] Exclude: .reviewboardrc
[INFO] Exclude: .git/**
[INFO] Exclude: .gitignore
[INFO] Exclude: .arcconfig
[INFO] Exclude: *.patch
[INFO] Exclude: .idea/**
[INFO] Exclude:**/*.iml
[INFO] Exclude:**/*.ipr
[INFO] Exclude:**/test/resources/**
[INFO] Exclude: giraph-gora/conf/*
[INFO] Exclude: giraph-rexster/giraph-kibble/src/main/resources/META-INF/services/com.tinkerpop.rexster.extension.RexsterExtension
[INFO] Exclude: src/site/resources/images/*.svg
[INFO] 
[INFO] >>> findbugs-maven-plugin:2.5.1:check (default) > :findbugs @ giraph-core >>>
[INFO] 
[INFO] --- findbugs-maven-plugin:2.5.1:findbugs (findbugs) @ giraph-core ---
[INFO] Fork Value is true
     [java] The following errors occurred during analysis:
     [java] Nov 17, 2015 3:16:36 PM edu.umd.cs.findbugs.TextUIBugReporter reportAnalysisError
     [java] SEVERE: Error scanning java/util/Comparator for referenced classes
     [java] java.lang.ArrayIndexOutOfBoundsException: 51966
     [java]     at org.objectweb.asm.ClassReader.readUTF8(Unknown Source)
     [java]     at org.objectweb.asm.ClassReader.readClass(Unknown Source)

.
.
.
.
.
.
.
.
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Giraph Parent ............................... SUCCESS [  4.140 s]
[INFO] Apache Giraph Core ................................. FAILURE [ 37.541 s]
[INFO] Apache Giraph Examples ............................. SKIPPED
[INFO] Apache Giraph Distribution ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 41.972 s
[INFO] Finished at: 2015-11-17T15:16:37+05:30
[INFO] Final Memory: 68M/537M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:findbugs-maven-plugin:2.5.1:findbugs (findbugs) on project giraph-core: Execution findbugs of goal org.codehaus.mojo:findbugs-maven-plugin:2.5.1:findbugs failed: Java returned: 1 -> [Help 1]
[ERROR]
JavahadoopMavengiraphBuild

2条答案

按热度按时间
sg24os4d

sg24os4d1#

我今天碰到了这个问题。最后我需要两个补丁来修复它：
在基本目录中打开pom.xml并替换 <munge.symbols>PURE_YARN,STATIC_SASL_SYMBOL</munge.symbols> 使用： <munge.symbols>PURE_YARN</munge.symbols> 应用此处找到的修补程序：giraph-1110.02.patch
使用 clean Maven旗： mvn -Phadoop_yarn -Dhadoop.version=2.7.3 clean package -DskipTests

赞(0）分享  回复(0）举报  2021-06-03
yizd12fk

yizd12fk2#

我今天早些时候遇到了这个问题，在邮件列表中找到了一个回复，建议从hadoop的munge.symbols中删除静态的\u sasl\u符号。
在做了那个改变之后，我就可以建立这个档案了。
以下是邮件列表存档的链接：http://mail-archives.apache.org/mod_mbox/giraph-user/201501.mbox/%3c54b17196.4040107@hiro-tan.org%3e

赞(0）分享  回复(0）举报  2021-06-03
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com