在ubuntu 22.04上构建hadoop-common 3.3.0，出现错误

k5ifujac  于 2023-10-15  发布在  Hadoop
关注(0)|答案(1)|浏览(259)

环境

下一篇：BUILDING.txt

Unix系统

  1. $ uname  
  2. Linux

JDK 1.8

  1. $ echo $JAVA_HOME  
  2. /usr/lib/jvm/java-8-openjdk-amd64/

Maven 3.3或更高版本

  1. $ mvn -v  
  2. Apache Maven 3.6.3  
  3. Maven home: /usr/share/maven  
  4. Java version: 1.8.0_382, vendor: Private Build, runtime: /usr/lib/jvm/java-8-openjdk-amd64/jre  
  5. Default locale: en_US, platform encoding: UTF-8  
  6. OS name: "linux", version: "6.2.0-32-generic", arch: "amd64", family: "unix"

Protocol Buffers 3.7.1（编译原生代码时）

  1. $ protoc --version  
  2. libprotoc 3.20.3

CMake 3.1或更高版本（如果编译本机代码）

  1. $ cmake --version  
  2. cmake version 3.22.1  
  3. CMake suite maintained and supported by Kitware (kitware.com/cmake).

Zlib devel（如果编译原生代码）

  1. $ python -c 'import zlib;print(zlib.ZLIB_VERSION)'  
  2. 1.2.13

Cyrus SASL devel（如果编译原生代码）

  1. $ saslpasswd2 -v  
  2. This product includes software developed by Computing Services  
  3. at Carnegie Mellon University (http://www.cmu.edu/computing/).  
  4. Built against SASL API version 2.1.27  
  5. LibSasl version 2.1.27 by "Cyrus SASL"

支持thread_local存储的编译器之一：GCC 4.8.1或更高版本、Visual Studio、Clang（社区版本）、Clang（iOS 9及更高版本）（如果编译本机代码）

  1. $ gcc --version  
  2. gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0  
  3. Copyright (C) 2021 Free Software Foundation, Inc.  
  4. This is free software; see the source for copying conditions.  There is NO  
  5. warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

openssl devel（如果编译本机hadoop-pipes并获得最佳HDFS加密性能）

  1. $ openssl version   
  2. OpenSSL 1.1.1u  30 May 2023

运行和执行

运行mvn clean install -DskipTests -Pdist,native后，出现异常：

  1. [WARNING] make[2]: Leaving directory '/home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/target/native'
  2. [WARNING] [ 38%] Linking C executable test_bulk_crc32
  3. [WARNING] /usr/bin/cmake -E cmake_link_script CMakeFiles/test_bulk_crc32.dir/link.txt --verbose=1
  4. [WARNING] /usr/bin/cc -std=gnu99  -g -O2 -Wall -pthread -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -rdynamic CMakeFiles/test_bulk_crc32.dir/main/native/src/org/apache/hadoop/util/bulk_crc32.c.o CMakeFiles/test_bulk_crc32.dir/main/native/src/org/apache/hadoop/util/bulk_crc32_x86.c.o CMakeFiles/test_bulk_crc32.dir/main/native/src/test/org/apache/hadoop/util/test_bulk_crc32.c.o -o test_bulk_crc32 
  5. [WARNING] make[2]: Leaving directory '/home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/target/native'
  6. [WARNING] [ 38%] Built target test_bulk_crc32
  7. [WARNING] make[2]: Leaving directory '/home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/target/native'
  8. [WARNING] make[1]: Leaving directory '/home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/target/native'
  9. [WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/exception.c: In function ‘terror’:
  10. [WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/exception.c:118:34: error: ‘sys_nerr’ undeclared (first use in this function)
  11. [WARNING]   118 |   if ((errnum < 0) || (errnum >= sys_nerr)) {
  12. [WARNING]       |                                  ^~~~~~~~
  13. [WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/exception.c:118:34: note: each undeclared identifier is reported only once for each function it appears in
  14. [WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/exception.c:121:10: error: ‘sys_errlist’ undeclared (first use in this function)
  15. [WARNING]   121 |   return sys_errlist[errnum];
  16. [WARNING]       |          ^~~~~~~~~~~
  17. [WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/exception.c:123:1: warning: control reaches end of non-void function [-Wreturn-type]
  18. [WARNING]   123 | }
  19. [WARNING]       | ^
  20. [WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/exception.c: In function ‘terror’:
  21. [WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/exception.c:118:34: error: ‘sys_nerr’ undeclared (first use in this function)
  22. [WARNING]   118 |   if ((errnum < 0) || (errnum >= sys_nerr)) {
  23. [WARNING]       |                                  ^~~~~~~~
  24. [WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/exception.c:118:34: note: each undeclared identifier is reported only once for each function it appears in
  25. [WARNING] make[2]: *** [CMakeFiles/hadoop.dir/build.make:76: CMakeFiles/hadoop.dir/main/native/src/exception.c.o] Error 1
  26. [WARNING] make[2]: *** Waiting for unfinished jobs....
  27. [WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/exception.c:121:10: error: ‘sys_errlist’ undeclared (first use in this function)
  28. [WARNING]   121 |   return sys_errlist[errnum];
  29. [WARNING]       |          ^~~~~~~~~~~
  30. [WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/exception.c:123:1: warning: control reaches end of non-void function [-Wreturn-type]
  31. [WARNING]   123 | }
  32. [WARNING]       | ^
  33. [WARNING] make[2]: *** [CMakeFiles/hadoop_static.dir/build.make:76: CMakeFiles/hadoop_static.dir/main/native/src/exception.c.o] Error 1
  34. [WARNING] make[2]: *** Waiting for unfinished jobs....
  35. [WARNING] make[1]: *** [CMakeFiles/Makefile2:113: CMakeFiles/hadoop_static.dir/all] Error 2
  36. [WARNING] make[1]: *** Waiting for unfinished jobs....
  37. [WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c: In function ‘locks_setup’:
  38. [WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:250:33: warning: implicit declaration of function ‘dlsym_CRYPTO_num_locks’; did you mean ‘dlsym_CRYPTO_malloc’? [-Wimplicit-function-declaration]
  39. [WARNING]   250 |   lock_cs = dlsym_CRYPTO_malloc(dlsym_CRYPTO_num_locks() *  \
  40. [WARNING]       |                                 ^~~~~~~~~~~~~~~~~~~~~~
  41. [WARNING]       |                                 dlsym_CRYPTO_malloc
  42. [WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:257:3: warning: implicit declaration of function ‘dlsym_CRYPTO_set_id_callback’; did you mean ‘CRYPTO_set_id_callback’? [-Wimplicit-function-declaration]
  43. [WARNING]   257 |   dlsym_CRYPTO_set_id_callback((unsigned long (*)())pthreads_thread_id);
  44. [WARNING]       |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
  45. [WARNING]       |   CRYPTO_set_id_callback
  46. [WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:258:3: warning: implicit declaration of function ‘dlsym_CRYPTO_set_locking_callback’; did you mean ‘CRYPTO_set_locking_callback’? [-Wimplicit-function-declaration]
  47. [WARNING]   258 |   dlsym_CRYPTO_set_locking_callback((void (*)())pthreads_locking_callback);
  48. [WARNING]       |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  49. [WARNING]       |   CRYPTO_set_locking_callback
  50. [WARNING] At top level:
  51. [WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:261:13: warning: ‘locks_cleanup’ defined but not used [-Wunused-function]
  52. [WARNING]   261 | static void locks_cleanup(void)
  53. [WARNING]       |             ^~~~~~~~~~~~~
  54. [WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:247:13: warning: ‘locks_setup’ defined but not used [-Wunused-function]
  55. [WARNING]   247 | static void locks_setup(void)
  56. [WARNING]       |             ^~~~~~~~~~~
  57. [WARNING] make[1]: *** [CMakeFiles/Makefile2:87: CMakeFiles/hadoop.dir/all] Error 2
  58. [WARNING] make: *** [Makefile:91: all] Error 2
  59. [INFO] ------------------------------------------------------------------------
  60. [INFO] BUILD FAILURE

有些人尝试

注意sys_errlist被标记为deprecated，因此将gcc降级到4.8.5，但会出现相同的错误。这里有我遗漏的依赖项吗？

hadoop

1条答案

按热度按时间
k2fxgqgv

k2fxgqgv1#

TL;DR

cherry-pick this commit or just use the following version：3.3.1，3.4.0，3.2.3，2.10.3

原因

很抱歉错过了this fixed！
最主要的原因是gcc的版本太高（对我来说是4.8.5）。
但是我在降级后得到了同样的错误，因为gcc链接到gcc-4.8，但是stdio.h总是使用系统库（说清楚，使用/usr/include/stdio.h），它已经删除了sys_errlist和sys_nerr！
有关更多信息，下面是<stdio.h>中包含的旧版本。

  1. 849 /* Provide the declarations for `sys_errlist' and `sys_nerr' if they
  2. 850    are available on this system.  Even if available, these variables
  3. 851    should not be used directly.  The `strerror' function provides
  4. 852    all the necessary functionality.  */
  5. 853 #include <bits/sys_errlist.h>

结论

错过this fixed有两个原因：

  • 谷歌它的关键字sys_nerr +hadoop和前3名是邮件存档，很难阅读，所以跳过它太快。
  • 缺乏C知识

首先意识到这是gcc版本的问题，但是用错误的方法修复了它，因为大多数答案只是建议apt install gcc-<some version>和update-alternative add，而没有标准库链接的提示。继续专注于gcc只是因为排除所有的部分，我是如此肯定，但只有gcc离开，因为我真的不擅长它。

展开查看全部
赞(0）分享  回复(0）举报  2023-10-15
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com