PaddleNLP [Bug]: undefined symbol: _ZN6paddle17OpMetaInfoBuilderC1EOSsm

eufgjt7s  于 5个月前  发布在  其他
关注(0)|答案(1)|浏览(71)

软件环境

paddle-bfloat              0.1.7
paddle2onnx                1.2.1
paddlefsl                  1.1.0
paddlehub                  2.4.0
paddlemix                  0.1.0               
paddlenlp                  3.0.0b0.post20240728
paddlepaddle-gpu           2.6.1.post112
paddleslim                 2.6.0

重复问题

  • 我已搜索现有的问题

错误描述

在编译csrc 算子时报错未定义的符号:_ZN6paddle17OpMetaInfoBuilderC1EOSsm
在相关问题中并未得到解决:
[按paddle教程自定义算子时链接库找不到](https://aistudio.baidu.com/paddle/forum/topic/show/1729464)
[#8222](https://github.com/PaddlePaddle/PaddleDetection/issues/8222)
[#9000](https://github.com/PaddlePaddle/PaddleDetection/issues/9000)
[#4045](https://github.com/PaddlePaddle/PaddleNLP/issues/4045)
[#43041](https://github.com/PaddlePaddle/Paddle/issues/43041)
其中[#8222]提到更换gcc版本,尝试过切换gcc 11.2 和gcc 8.5 无效

### 报错log输出

[2024-07-31 13:40:37,079] [    INFO] file_util.py:130 - copying build/paddlenlp_ops/lib.linux-x86_64-cpython-39/write_cache_kv.cu.o -> build/paddlenlp_ops/bdist.linux-x86_64/egg        
[2024-07-31 13:40:37,082] [    INFO] file_util.py:130 - copying build/paddlenlp_ops/lib.linux-x86_64-cpython-39/paddlenlp_ops.so -> build/paddlenlp_ops/bdist.linux-x86_64/egg           
[2024-07-31 13:40:37,099] [    INFO] file_util.py:130 - copying build/paddlenlp_ops/lib.linux-x86_64-cpython-39/dequant_int8.cu.o -> build/paddlenlp_ops/bdist.linux-x86_64/egg          
[2024-07-31 13:40:37,103] [    INFO] file_util.py:130 - copying build/paddlenlp_ops/lib.linux-x86_64-cpython-39/transpose_removing_padding.cu.o -> build/paddlenlp_ops/bdist.linux-x86_64
/egg                                                                                                                                                                                     
[2024-07-31 13:40:37,108] [    INFO] file_util.py:130 - copying build/paddlenlp_ops/lib.linux-x86_64-cpython-39/save_with_output_msg.o -> build/paddlenlp_ops/bdist.linux-x86_64/egg     
[2024-07-31 13:40:37,183] [    INFO] bdist_egg.py:177 - creating stub loader for paddlenlp_ops.so                                                                                        
Traceback (most recent call last):                                                                                                                                                       
  File "/mnt/petrelfs/taitan/projects/PaddleMIX/PaddleNLP/csrc/setup_cuda.py", line 53, in <module>                                                                                     
    setup(                                                                                                                                                                               
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/paddle/utils/cpp_extension/cpp_extension.py", line 236, in setup                                        
    setuptools.setup(**attr)                                                                                                                                                             
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/setuptools/__init__.py", line 104, in setup                                                             
    return distutils.core.setup(**attrs)                                                                                                                                                 
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 184, in setup
    return run_commands(dist)
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
    dist.run_commands()
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
    self.run_command(cmd)
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/setuptools/dist.py", line 967, in run_command    super().run_command(command)  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/setuptools/command/install.py", line 87, in run
    self.do_egg_install()
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/setuptools/command/install.py", line 139, in do_egg_install
    self.run_command('bdist_egg')
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
    self.distribution.run_command(command)
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/setuptools/dist.py", line 967, in run_command
    super().run_command(command)
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/setuptools/command/bdist_egg.py", line 179, in run
    write_stub(os.path.basename(ext_name), pyfile)
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/paddle/utils/cpp_extension/extension_utils.py", line 222, in custom_write_stub
    new_custom_ops = load_op_meta_info_and_register_op(so_path)
  File "/mnt/petrelfs/taitan/miniconda3/envs/paddle/lib/python3.9/site-packages/paddle/utils/cpp_extension/extension_utils.py", line 160, in load_op_meta_info_and_register_op
    core.load_op_meta_info_and_register_op(lib_filename)
RuntimeError: (PreconditionNotMet) The third-party dynamic library (/mnt/petrelfs/taitan/projects/PaddleMIX/PaddleNLP/csrc/build/paddlenlp_ops/lib.linux-x86_64-cpython-39/paddlenlp_ops
.so) that Paddle depends on is not configured correctly. (error code is /mnt/petrelfs/taitan/projects/PaddleMIX/PaddleNLP/csrc/build/paddlenlp_ops/lib.linux-x86_64-cpython-39/paddlenlp
_ops.so: undefined symbol: _ZN6paddle17OpMetaInfoBuilderC1EOSsm)
  Suggestions:
  1. Check if the third-party dynamic library (e.g. CUDA, CUDNN) is installed correctly and its version is matched with paddlepaddle you installed.
  2. Configure third-party dynamic library environment variables as follows:
  - Linux: set LD_LIBRARY_PATH by `export LD_LIBRARY_PATH=...`
  - Windows: set PATH by `set PATH=XXX; (at ../paddle/phi/backends/dynload/dynamic_loader.cc:312)
### 稳定复现步骤 & 代码

Ubuntu 20.04.6 LTS
A100 80G
cuda 11.2
gcc 8.5.0
python=3.9.19

cudnn
#define CUDNN_MAJOR 8
#define CUDNN_MINOR 1
#define CUDNN_PATCHLEVEL 1
hivapdat

hivapdat1#

这个问题是由系统环境引起的,可能在不同的环境中有不同的原因。下面是一个检查方案:

  • 检查cmake版本,并与官方的docker环境进行对照。
  • 检查系统中是否存在python库,因为在编译过程中会用到python.h文件,这个文件在conda中可能不存在。因此,需要链接到系统的python库。可以通过检查编译过程的输出来查看。
  • 检查gcc版本,v8以上的版本应该都可以正常工作。

相关问题