在ubuntu 22.04上构建hadoop-common 3.3.0，出现错误

k5ifujac  于 2023-10-15  发布在  Hadoop
关注(0)|答案(1)|浏览(208)

环境

下一篇：BUILDING.txt

Unix系统

$ uname  
Linux

JDK 1.8

$ echo $JAVA_HOME  
/usr/lib/jvm/java-8-openjdk-amd64/

Maven 3.3或更高版本

$ mvn -v  
Apache Maven 3.6.3  
Maven home: /usr/share/maven  
Java version: 1.8.0_382, vendor: Private Build, runtime: /usr/lib/jvm/java-8-openjdk-amd64/jre  
Default locale: en_US, platform encoding: UTF-8  
OS name: "linux", version: "6.2.0-32-generic", arch: "amd64", family: "unix"

Protocol Buffers 3.7.1（编译原生代码时）

$ protoc --version  
libprotoc 3.20.3

CMake 3.1或更高版本（如果编译本机代码）

$ cmake --version  
cmake version 3.22.1  
CMake suite maintained and supported by Kitware (kitware.com/cmake).

Zlib devel（如果编译原生代码）

$ python -c 'import zlib;print(zlib.ZLIB_VERSION)'  
1.2.13

Cyrus SASL devel（如果编译原生代码）

$ saslpasswd2 -v  
This product includes software developed by Computing Services  
at Carnegie Mellon University (http://www.cmu.edu/computing/).  
Built against SASL API version 2.1.27  
LibSasl version 2.1.27 by "Cyrus SASL"

支持thread_local存储的编译器之一：GCC 4.8.1或更高版本、Visual Studio、Clang（社区版本）、Clang（iOS 9及更高版本）（如果编译本机代码）

$ gcc --version  
gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0  
Copyright (C) 2021 Free Software Foundation, Inc.  
This is free software; see the source for copying conditions.  There is NO  
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

openssl devel（如果编译本机hadoop-pipes并获得最佳HDFS加密性能）

$ openssl version   
OpenSSL 1.1.1u  30 May 2023

运行和执行

运行mvn clean install -DskipTests -Pdist,native后，出现异常：

[WARNING] make[2]: Leaving directory '/home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/target/native'
[WARNING] [ 38%] Linking C executable test_bulk_crc32
[WARNING] /usr/bin/cmake -E cmake_link_script CMakeFiles/test_bulk_crc32.dir/link.txt --verbose=1
[WARNING] /usr/bin/cc -std=gnu99  -g -O2 -Wall -pthread -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -rdynamic CMakeFiles/test_bulk_crc32.dir/main/native/src/org/apache/hadoop/util/bulk_crc32.c.o CMakeFiles/test_bulk_crc32.dir/main/native/src/org/apache/hadoop/util/bulk_crc32_x86.c.o CMakeFiles/test_bulk_crc32.dir/main/native/src/test/org/apache/hadoop/util/test_bulk_crc32.c.o -o test_bulk_crc32 
[WARNING] make[2]: Leaving directory '/home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/target/native'
[WARNING] [ 38%] Built target test_bulk_crc32
[WARNING] make[2]: Leaving directory '/home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/target/native'
[WARNING] make[1]: Leaving directory '/home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/target/native'
[WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/exception.c: In function ‘terror’:
[WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/exception.c:118:34: error: ‘sys_nerr’ undeclared (first use in this function)
[WARNING]   118 |   if ((errnum < 0) || (errnum >= sys_nerr)) {
[WARNING]       |                                  ^~~~~~~~
[WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/exception.c:118:34: note: each undeclared identifier is reported only once for each function it appears in
[WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/exception.c:121:10: error: ‘sys_errlist’ undeclared (first use in this function)
[WARNING]   121 |   return sys_errlist[errnum];
[WARNING]       |          ^~~~~~~~~~~
[WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/exception.c:123:1: warning: control reaches end of non-void function [-Wreturn-type]
[WARNING]   123 | }
[WARNING]       | ^
[WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/exception.c: In function ‘terror’:
[WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/exception.c:118:34: error: ‘sys_nerr’ undeclared (first use in this function)
[WARNING]   118 |   if ((errnum < 0) || (errnum >= sys_nerr)) {
[WARNING]       |                                  ^~~~~~~~
[WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/exception.c:118:34: note: each undeclared identifier is reported only once for each function it appears in
[WARNING] make[2]: *** [CMakeFiles/hadoop.dir/build.make:76: CMakeFiles/hadoop.dir/main/native/src/exception.c.o] Error 1
[WARNING] make[2]: *** Waiting for unfinished jobs....
[WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/exception.c:121:10: error: ‘sys_errlist’ undeclared (first use in this function)
[WARNING]   121 |   return sys_errlist[errnum];
[WARNING]       |          ^~~~~~~~~~~
[WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/exception.c:123:1: warning: control reaches end of non-void function [-Wreturn-type]
[WARNING]   123 | }
[WARNING]       | ^
[WARNING] make[2]: *** [CMakeFiles/hadoop_static.dir/build.make:76: CMakeFiles/hadoop_static.dir/main/native/src/exception.c.o] Error 1
[WARNING] make[2]: *** Waiting for unfinished jobs....
[WARNING] make[1]: *** [CMakeFiles/Makefile2:113: CMakeFiles/hadoop_static.dir/all] Error 2
[WARNING] make[1]: *** Waiting for unfinished jobs....
[WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c: In function ‘locks_setup’:
[WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:250:33: warning: implicit declaration of function ‘dlsym_CRYPTO_num_locks’; did you mean ‘dlsym_CRYPTO_malloc’? [-Wimplicit-function-declaration]
[WARNING]   250 |   lock_cs = dlsym_CRYPTO_malloc(dlsym_CRYPTO_num_locks() *  \
[WARNING]       |                                 ^~~~~~~~~~~~~~~~~~~~~~
[WARNING]       |                                 dlsym_CRYPTO_malloc
[WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:257:3: warning: implicit declaration of function ‘dlsym_CRYPTO_set_id_callback’; did you mean ‘CRYPTO_set_id_callback’? [-Wimplicit-function-declaration]
[WARNING]   257 |   dlsym_CRYPTO_set_id_callback((unsigned long (*)())pthreads_thread_id);
[WARNING]       |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
[WARNING]       |   CRYPTO_set_id_callback
[WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:258:3: warning: implicit declaration of function ‘dlsym_CRYPTO_set_locking_callback’; did you mean ‘CRYPTO_set_locking_callback’? [-Wimplicit-function-declaration]
[WARNING]   258 |   dlsym_CRYPTO_set_locking_callback((void (*)())pthreads_locking_callback);
[WARNING]       |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[WARNING]       |   CRYPTO_set_locking_callback
[WARNING] At top level:
[WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:261:13: warning: ‘locks_cleanup’ defined but not used [-Wunused-function]
[WARNING]   261 | static void locks_cleanup(void)
[WARNING]       |             ^~~~~~~~~~~~~
[WARNING] /home/x/workspace/dp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/random/OpensslSecureRandom.c:247:13: warning: ‘locks_setup’ defined but not used [-Wunused-function]
[WARNING]   247 | static void locks_setup(void)
[WARNING]       |             ^~~~~~~~~~~
[WARNING] make[1]: *** [CMakeFiles/Makefile2:87: CMakeFiles/hadoop.dir/all] Error 2
[WARNING] make: *** [Makefile:91: all] Error 2
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE

有些人尝试

注意sys_errlist被标记为deprecated，因此将gcc降级到4.8.5，但会出现相同的错误。这里有我遗漏的依赖项吗？

hadoop

1条答案

按热度按时间
k2fxgqgv

k2fxgqgv1#

TL;DR

cherry-pick this commit or just use the following version：3.3.1，3.4.0，3.2.3，2.10.3

原因

很抱歉错过了this fixed！
最主要的原因是gcc的版本太高（对我来说是4.8.5）。
但是我在降级后得到了同样的错误，因为gcc链接到gcc-4.8，但是stdio.h总是使用系统库（说清楚，使用/usr/include/stdio.h），它已经删除了sys_errlist和sys_nerr！
有关更多信息，下面是<stdio.h>中包含的旧版本。

849 /* Provide the declarations for `sys_errlist' and `sys_nerr' if they
850    are available on this system.  Even if available, these variables
851    should not be used directly.  The `strerror' function provides
852    all the necessary functionality.  */
853 #include <bits/sys_errlist.h>

结论

错过this fixed有两个原因：

  • 谷歌它的关键字sys_nerr +hadoop和前3名是邮件存档，很难阅读，所以跳过它太快。
  • 缺乏C知识

首先意识到这是gcc版本的问题，但是用错误的方法修复了它，因为大多数答案只是建议apt install gcc-<some version>和update-alternative add，而没有标准库链接的提示。继续专注于gcc只是因为排除所有的部分，我是如此肯定，但只有gcc离开，因为我真的不擅长它。

赞(0）分享  回复(0）举报  2023-10-15
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com