我尝试在dev hadoop群集上安装和配置apache airflow,该群集由三个节点组成,配置/版本如下:
Operating System: Red Hat Enterprise Linux Server 7.7
python 3.7.3
anaconda 2
spark 2.45
a)sudo yum install gcc gcc-c++ -y
b)sudo yum install libffi-devel mariadb-devel cyrus-sasl-devel -y
c)pip install 'apache-airflow[all]'
d)airflow initdb -- airflow.cfgfile was created with SQLlite
然后我按照下面的一组命令用mysql对其进行配置
a) rpm -Uvh https://repo.mysql.com/mysql80-community-release-el7-3.noarch.rpm
b) sed -i 's/enabled=1/enabled=0/' /etc/yum.repos.d/mysql-community.repo
c) yum --enablerepo=mysql80-community install mysql-community-server
d) systemctl start mysqld.service
在mysql做了以下事情
a) CREATE DATABASE airflow CHARACTER SET utf8 COLLATE utf8_unicode_ci;
b) create user 'airflow'@'localhost' identified by 'Airflow123';
c) grant all privileges on * . * to 'airflow'@'localhost';
以下是我的一些细节 airflow.cfg
文件
broker_url = sqla+mysql://airflow:airflow@localhost:3306/airflow
result_backend = db+mysql://airflow:airflow@localhost:3306/airflow
sql_alchemy_conn = mysql://airflow:Airflow123@localhost:3306/airflow
executor = CeleryExecutor
我在跑步时出错了 airflow initdb
命令
ImportError: /home/xyz/anaconda2/envs/python3.7.2/lib/python3.7/site-packages/_mysql.cpython-37m-x86_64-linux-gnu.so: symbol mysql_real_escape_string_quote,
version libmysqlclient_18 not defined in file libmysqlclient.so.18 with link time reference
已将.bashrc文件设置为:
export AIRFLOW_HOME=~/airflow
以下是我创建的目录:
[xyz@innolx5984 airflow]$ pwd
/home/xyz/airflow
当我查找此文件时“ libmysqlclient
“我发现了很多这样的例子。
[xyz@innolx5984 airflow]$ find /home/xyz/ -name "*libmysqlclient*"
/home/xyz/anaconda2/pkgs/mysql-connector-c-6.1.11-h597af5e_1/lib/libmysqlclient.so
/home/xyz/anaconda2/pkgs/mysql-connector-c-6.1.11-h597af5e_1/lib/libmysqlclient.a
/home/xyz/anaconda2/pkgs/mysql-connector-c-6.1.11-h597af5e_1/lib/libmysqlclient.so.18
/home/xyz/anaconda2/pkgs/mysql-connector-c-6.1.11-h597af5e_1/lib/libmysqlclient.so.18.4.0
/home/xyz/anaconda2/lib/libmysqlclient.a
/home/xyz/anaconda2/lib/libmysqlclient.so
/home/xyz/anaconda2/lib/libmysqlclient.so.18
/home/xyz/anaconda2/lib/libmysqlclient.so.18.4.0
只是增加一些细节以防万一。
[xyz@innolx5984 airflow]$ mysql_config
Usage: /home/xyz/an
aconda2/bin/mysql_config [OPTIONS]
Options:
--cflags [-I/home/xyz/anaconda2/include ]
--cxxflags [-I/home/xyz/anaconda2/include ]
--include [-I/home/xyz/anaconda2/include]
--libs [-L/home/xyz/anaconda2/lib -lmysqlclient ]
--libs_r [-L/home/xyz/anaconda2/lib -lmysqlclient ]
--plugindir [/home/xyz`/anaconda2/lib/plugin]
--socket [/tmp/mysql.sock]
--port [0]
--version [6.1.11]
--variable=VAR VAR is one of:
pkgincludedir [/home/xyz/anaconda2/include]
pkglibdir [/home/xyz/anaconda2/lib]
plugindir [/home/xyz/anaconda2/lib/plugin]
Looking for some help and suggestion to resolve this
问题。我不太确定方向是否正确。
1条答案
按热度按时间edqdpe6u1#
按照以下步骤使用anaconda3通过mysql安装apache aiffort
1) 安装先决条件
2) 安装anaconda3(随python 3.7.6提供)
一定要这么做
conda initialize
在安装过程中出现提示时。这将确保在后续步骤中使用正确版本的python和pip。3) 安装apache airflow
您可以根据需要添加其他子包。我只包括了使用mysql数据库作为后端所需的那些。
4) 初始化气流
从这里开始,我模仿了您配置mysql服务器所遵循的步骤
5) 安装mysql服务器
6) 登录mysql并配置数据库
7) 更新气流配置文件(~/airflow/airflow.cfg)
8) 初始化气流