在hadoop中使用apache airflow配置mysql时面临Python3.x问题

cuxqih21  于 2021-05-31  发布在  Hadoop
关注(0)|答案(1)|浏览(408)

我尝试在dev hadoop群集上安装和配置apache airflow,该群集由三个节点组成,配置/版本如下:

  1. Operating System: Red Hat Enterprise Linux Server 7.7
  2. python 3.7.3
  3. anaconda 2
  4. spark 2.45
  5. a)sudo yum install gcc gcc-c++ -y
  6. b)sudo yum install libffi-devel mariadb-devel cyrus-sasl-devel -y
  7. c)pip install 'apache-airflow[all]'
  8. d)airflow initdb -- airflow.cfgfile was created with SQLlite

然后我按照下面的一组命令用mysql对其进行配置

  1. a) rpm -Uvh https://repo.mysql.com/mysql80-community-release-el7-3.noarch.rpm
  2. b) sed -i 's/enabled=1/enabled=0/' /etc/yum.repos.d/mysql-community.repo
  3. c) yum --enablerepo=mysql80-community install mysql-community-server
  4. d) systemctl start mysqld.service

在mysql做了以下事情

  1. a) CREATE DATABASE airflow CHARACTER SET utf8 COLLATE utf8_unicode_ci;
  2. b) create user 'airflow'@'localhost' identified by 'Airflow123';
  3. c) grant all privileges on * . * to 'airflow'@'localhost';

以下是我的一些细节 airflow.cfg 文件

  1. broker_url = sqla+mysql://airflow:airflow@localhost:3306/airflow
  2. result_backend = db+mysql://airflow:airflow@localhost:3306/airflow
  3. sql_alchemy_conn = mysql://airflow:Airflow123@localhost:3306/airflow
  4. executor = CeleryExecutor

我在跑步时出错了 airflow initdb 命令

  1. ImportError: /home/xyz/anaconda2/envs/python3.7.2/lib/python3.7/site-packages/_mysql.cpython-37m-x86_64-linux-gnu.so: symbol mysql_real_escape_string_quote,
  2. version libmysqlclient_18 not defined in file libmysqlclient.so.18 with link time reference

已将.bashrc文件设置为:

  1. export AIRFLOW_HOME=~/airflow

以下是我创建的目录:

  1. [xyz@innolx5984 airflow]$ pwd
  2. /home/xyz/airflow

当我查找此文件时“ libmysqlclient “我发现了很多这样的例子。

  1. [xyz@innolx5984 airflow]$ find /home/xyz/ -name "*libmysqlclient*"
  2. /home/xyz/anaconda2/pkgs/mysql-connector-c-6.1.11-h597af5e_1/lib/libmysqlclient.so
  3. /home/xyz/anaconda2/pkgs/mysql-connector-c-6.1.11-h597af5e_1/lib/libmysqlclient.a
  4. /home/xyz/anaconda2/pkgs/mysql-connector-c-6.1.11-h597af5e_1/lib/libmysqlclient.so.18
  5. /home/xyz/anaconda2/pkgs/mysql-connector-c-6.1.11-h597af5e_1/lib/libmysqlclient.so.18.4.0
  6. /home/xyz/anaconda2/lib/libmysqlclient.a
  7. /home/xyz/anaconda2/lib/libmysqlclient.so
  8. /home/xyz/anaconda2/lib/libmysqlclient.so.18
  9. /home/xyz/anaconda2/lib/libmysqlclient.so.18.4.0

只是增加一些细节以防万一。

  1. [xyz@innolx5984 airflow]$ mysql_config
  2. Usage: /home/xyz/an
  3. aconda2/bin/mysql_config [OPTIONS]
  4. Options:
  5. --cflags [-I/home/xyz/anaconda2/include ]
  6. --cxxflags [-I/home/xyz/anaconda2/include ]
  7. --include [-I/home/xyz/anaconda2/include]
  8. --libs [-L/home/xyz/anaconda2/lib -lmysqlclient ]
  9. --libs_r [-L/home/xyz/anaconda2/lib -lmysqlclient ]
  10. --plugindir [/home/xyz`/anaconda2/lib/plugin]
  11. --socket [/tmp/mysql.sock]
  12. --port [0]
  13. --version [6.1.11]
  14. --variable=VAR VAR is one of:
  15. pkgincludedir [/home/xyz/anaconda2/include]
  16. pkglibdir [/home/xyz/anaconda2/lib]
  17. plugindir [/home/xyz/anaconda2/lib/plugin]
  18. Looking for some help and suggestion to resolve this

问题。我不太确定方向是否正确。

edqdpe6u

edqdpe6u1#

按照以下步骤使用anaconda3通过mysql安装apache aiffort
1) 安装先决条件

  1. yum install gcc gcc-c++ -y
  2. yum install libffi-devel mariadb-devel cyrus-sasl-devel -y
  3. dnf install redhat-rpm-config

2) 安装anaconda3(随python 3.7.6提供)

  1. yum install libXcomposite libXcursor libXi libXtst libXrandr alsa-lib mesa-libEGL libXdamage mesa-libGL libXScrnSaver
  2. wget https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh
  3. chmod +x Anaconda3-2020.02-Linux-x86_64.sh
  4. ./Anaconda3-2020.02-Linux-x86_64.sh

一定要这么做 conda initialize 在安装过程中出现提示时。这将确保在后续步骤中使用正确版本的python和pip。
3) 安装apache airflow

  1. pip install apache-airflow[mysql,celery]

您可以根据需要添加其他子包。我只包括了使用mysql数据库作为后端所需的那些。
4) 初始化气流

  1. export AIRFLOW_HOME=~/airflow
  2. airflow initdb

从这里开始,我模仿了您配置mysql服务器所遵循的步骤
5) 安装mysql服务器

  1. rpm -Uvh https://repo.mysql.com/mysql80-community-release-el7-3.noarch.rpm
  2. sed -i 's/enabled=1/enabled=0/' /etc/yum.repos.d/mysql-community.repo
  3. yum --enablerepo=mysql80-community install mysql-server
  4. systemctl start mysqld.service

6) 登录mysql并配置数据库

  1. mysql> CREATE DATABASE airflow CHARACTER SET utf8 COLLATE utf8_unicode_ci;
  2. mysql> CREATE user 'airflow'@'localhost' identified by 'Airflow123';
  3. mysql> GRANT ALL privileges on *.* to 'airflow'@'localhost';

7) 更新气流配置文件(~/airflow/airflow.cfg)

  1. sql_alchemy_conn = mysql://airflow:Airflow123@localhost:3306/airflow
  2. executor = CeleryExecutor

8) 初始化气流

  1. airflow initdb
展开查看全部

相关问题