java—在mac os上从源代码编译hadoop本机libs

v8wbuo2f  于 2021-05-29  发布在  Hadoop
关注(0)|答案(2)|浏览(448)

我正在努力从源代码处编译最新的hadoop，下一个例外是：

  1. [INFO] Executed tasks
  2. [INFO]
  3. [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hadoop-pipes ---
  4. [INFO]
  5. [INFO] --- hadoop-maven-plugins:3.0.0-alpha2-SNAPSHOT:cmake-compile (cmake-compile) @ hadoop-pipes ---
  6. [INFO] Running cmake /Users/romeo/Repository/hadoop/hadoop-tools/hadoop-pipes/src -DJVM_ARCH_DATA_MODEL=64 -G Unix Makefiles
  7. [INFO] with extra environment variables {}
  8. [WARNING] CMake Error at /usr/local/Cellar/cmake/3.7.0/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:138 (message):
  9. [WARNING]   Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the
  10. [WARNING]   system variable OPENSSL_ROOT_DIR (missing: OPENSSL_INCLUDE_DIR)
  11. [WARNING] Call Stack (most recent call first):
  12. [WARNING]   /usr/local/Cellar/cmake/3.7.0/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
  13. [WARNING]   /usr/local/Cellar/cmake/3.7.0/share/cmake/Modules/FindOpenSSL.cmake:385 (find_package_handle_standard_args)
  14. [WARNING]   CMakeLists.txt:24 (find_package)
  15. [WARNING]
  16. [WARNING]
  17. [WARNING] -- Configuring incomplete, errors occurred!
  18. [WARNING] See also "/Users/romeo/Repository/hadoop/hadoop-tools/hadoop-pipes/target/native/CMakeFiles/CMakeOutput.log".
  19. [INFO] ------------------------------------------------------------------------
  20. [INFO] Reactor Summary:
  21. [INFO]
  22. [INFO] Apache Hadoop Main ................................. SUCCESS [  1.920 s]
  23. [INFO] Apache Hadoop Build Tools .......................... SUCCESS [  0.939 s]
  24. [INFO] Apache Hadoop Project POM .......................... SUCCESS [  1.191 s]
  25. [INFO] Apache Hadoop Annotations .......................... SUCCESS [  2.458 s]
  26. [INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.170 s]
  27. [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  2.170 s]
  28. [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  6.001 s]
  29. [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  2.502 s]
  30. [INFO] Apache Hadoop Auth ................................. SUCCESS [  7.454 s]
  31. [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  4.214 s]
  32. [INFO] Apache Hadoop Common ............................... SUCCESS [ 56.405 s]
  33. [INFO] Apache Hadoop NFS .................................. SUCCESS [  5.317 s]
  34. [INFO] Apache Hadoop KMS .................................. SUCCESS [  7.725 s]
  35. [INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.044 s]
  36. [INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 15.292 s]
  37. [INFO] Apache Hadoop HDFS ................................. SUCCESS [ 39.539 s]
  38. [INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [  2.075 s]
  39. [INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 22.573 s]
  40. [INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  3.311 s]
  41. [INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.049 s]
  42. [INFO] Apache Hadoop YARN ................................. SUCCESS [  0.044 s]
  43. [INFO] Apache Hadoop YARN API ............................. SUCCESS [  8.825 s]
  44. [INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 22.424 s]
  45. [INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.030 s]
  46. [INFO] Apache Hadoop YARN Server Common ................... SUCCESS [  5.927 s]
  47. [INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [  8.990 s]
  48. [INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  2.795 s]
  49. [INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [  4.752 s]
  50. [INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [  7.784 s]
  51. [INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 14.938 s]
  52. [INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [  1.469 s]
  53. [INFO] Apache Hadoop YARN Client .......................... SUCCESS [  4.031 s]
  54. [INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [  2.766 s]
  55. [INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [  2.808 s]
  56. [INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [  2.918 s]
  57. [INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.041 s]
  58. [INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  2.676 s]
  59. [INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  2.113 s]
  60. [INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.043 s]
  61. [INFO] Apache Hadoop YARN Registry ........................ SUCCESS [  3.960 s]
  62. [INFO] Apache Hadoop YARN UI .............................. SUCCESS [  0.029 s]
  63. [INFO] Apache Hadoop YARN Project ......................... SUCCESS [  6.991 s]
  64. [INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  0.138 s]
  65. [INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 14.384 s]
  66. [INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 12.110 s]
  67. [INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [  3.040 s]
  68. [INFO] Apache Hadoop MapReduce App ........................ SUCCESS [  6.127 s]
  69. [INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [  4.264 s]
  70. [INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [  3.597 s]
  71. [INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  1.715 s]
  72. [INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [  6.028 s]
  73. [INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  4.248 s]
  74. [INFO] Apache Hadoop MapReduce ............................ SUCCESS [  3.660 s]
  75. [INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  3.836 s]
  76. [INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  3.238 s]
  77. [INFO] Apache Hadoop Archives ............................. SUCCESS [  1.621 s]
  78. [INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  2.315 s]
  79. [INFO] Apache Hadoop Rumen ................................ SUCCESS [  4.120 s]
  80. [INFO] Apache Hadoop Gridmix .............................. SUCCESS [  2.797 s]
  81. [INFO] Apache Hadoop Data Join ............................ SUCCESS [  1.888 s]
  82. [INFO] Apache Hadoop Extras ............................... SUCCESS [  1.628 s]
  83. [INFO] Apache Hadoop Pipes ................................ FAILURE [  0.533 s]
  84. [INFO] Apache Hadoop OpenStack support .................... SKIPPED
  85. [INFO] Apache Hadoop Amazon Web Services support .......... SKIPPED
  86. [INFO] Apache Hadoop Azure support ........................ SKIPPED
  87. [INFO] Apache Hadoop Aliyun OSS support ................... SKIPPED
  88. [INFO] Apache Hadoop Client ............................... SKIPPED
  89. [INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED
  90. [INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED
  91. [INFO] Apache Hadoop Azure Data Lake support .............. SKIPPED
  92. [INFO] Apache Hadoop Tools Dist ........................... SKIPPED
  93. [INFO] Apache Hadoop Kafka Library support ................ SKIPPED
  94. [INFO] Apache Hadoop Tools ................................ SKIPPED
  95. [INFO] Apache Hadoop Distribution ......................... SKIPPED
  96. [INFO] ------------------------------------------------------------------------
  97. [INFO] BUILD FAILURE
  98. [INFO] ------------------------------------------------------------------------
  99. [INFO] Total time: 06:12 min
  100. [INFO] Finished at: 2016-11-26T07:18:34+01:00
  101. [INFO] Final Memory: 146M/857M
  102. [INFO] ------------------------------------------------------------------------
  103. [ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.0.0-alpha2-SNAPSHOT:cmake-compile (cmake-compile) on project hadoop-pipes: CMake failed with error code 1 -> [Help 1]
  104. [ERROR]

从输出中可以看到，由于未解析的依赖关系“apache hadoop maven plugins”，它在“apache hadoop pipes”处失败。但是，maven插件的状态是成功的：

  1. [ERROR] Plugin org.apache.hadoop:hadoop-maven-plugins:3.0.0-alpha2-SNAPSHOT or one of its dependencies could not be resolved: Could not find artifact org.apache.hadoop:hadoop-maven-plugins:jar:3.0.0-alpha2-SNAPSHOT -> [Help 1]
  2. org.apache.maven.plugin.PluginResolutionException: Plugin org.apache.hadoop:hadoop-maven-plugins:3.0.0-alpha2-SNAPSHOT or one of its dependencies could not be resolved: Could not find artifact org.apache.hadoop:hadoop-maven-plugins:jar:3.0.0-alpha2-SNAPSHOT
  3.     at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve(DefaultPluginDependenciesResolver.java:128)
  4.     at org.apache.maven.plugin.internal.DefaultMavenPluginManager.getPluginDescriptor(DefaultMavenPluginManager.java:179)
  5.     at org.apache.maven.plugin.internal.DefaultMavenPluginManager.getMojoDescriptor(DefaultMavenPluginManager.java:284)
  6.     at org.apache.maven.plugin.DefaultBuildPluginManager.getMojoDescriptor(DefaultBuildPluginManager.java:241)
  7.     at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator.setupMojoExecution(DefaultLifecycleExecutionPlanCalculator.java:169)
  8.     at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator.setupMojoExecutions(DefaultLifecycleExecutionPlanCalculator.java:155)
  9.     at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator.calculateExecutionPlan(DefaultLifecycleExecutionPlanCalculator.java:131)
  10.     at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator.calculateExecutionPlan(DefaultLifecycleExecutionPlanCalculator.java:145)
  11.     at org.apache.maven.lifecycle.internal.builder.BuilderCommon.resolveBuildPlan(BuilderCommon.java:96)
  12.     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:109)
  13.     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
  14.     at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
  15.     at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
  16.     at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
  17.     at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
  18.     at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
  19.     at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
  20.     at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
  21.     at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
  22.     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  23.     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  24.     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  25.     at java.lang.reflect.Method.invoke(Method.java:497)
  26.     at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
  27.     at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
  28.     at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
  29.     at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
  30. Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not find artifact org.apache.hadoop:hadoop-maven-plugins:jar:3.0.0-alpha2-SNAPSHOT
  31.     at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:444)
  32.     at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:246)
  33.     at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:223)
  34.     at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveArtifact(DefaultRepositorySystem.java:294)
  35.     at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve(DefaultPluginDependenciesResolver.java:124)
  36.     ... 26 more
  37. Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact org.apache.hadoop:hadoop-maven-plugins:jar:3.0.0-alpha2-SNAPSHOT
  38.     at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:434)
  39.     ... 30 more
  40. [ERROR]

任何建议或暗示都将不胜感激。谢谢。
更新1:安装openssl确实解决了主要问题：

  1. [INFO] ------------------------------------------------------------------------
  2. [INFO] Building Apache Hadoop Pipes 3.0.0-alpha2-SNAPSHOT
  3. [INFO] ------------------------------------------------------------------------
  4. [WARNING] The POM for org.apache.hadoop:hadoop-maven-plugins:jar:3.0.0-alpha2-SNAPSHOT is missing, no dependency information available

更新2:执行后 mvn install 获取新异常：

  1. [ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.0.0-alpha2-SNAPSHOT:cmake-compile (cmake-compile) on project hadoop-pipes: CMake failed with error code 1 -> [Help 1]
  2. org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.0.0-alpha2-SNAPSHOT:cmake-compile (cmake-compile) on project hadoop-pipes: CMake failed with error code 1
  3.     at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
  4.     at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
  5.     at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
JavahadoopMavenmacoscmake

2条答案

按热度按时间
2vuwiymt

2vuwiymt1#

答案就在这里
问题与openssl包有关。默认情况下，它不包括dev头。

赞(0）分享  回复(0）举报  2021-05-29
14ifxucb

14ifxucb2#

安装 hadoop-maven-plugin 可以通过 mvn install 第一。

  1. $ mvn clean install -DskipTests
  2. $ mvn package -Pdist -Pnative -Dtar -DskipTests

依赖项详细信息在building.txt中描述。
或者如果你有docker环境， start-build-env.sh 创建可以在其上构建hadoop的docker容器。

  1. $ cd hadoop
  2. $ ./start-build-env.sh # Automatically login

由于所有必需的依赖项都已安装在此容器中，因此可以在该容器中构建包含本机的hadoop包。

赞(0）分享  回复(0）举报  2021-05-29
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com