flume

yvt65v4c  于 2021-06-03  发布在  Hadoop
关注(0)|答案(3)|浏览(271)

我想在ubuntu上安装apacheflume 1.4.0来移动数据并将其存储到hdfs中,但是我找不到显示如何正确安装flume的安装指南。需要帮忙吗?

9lowa7mx

9lowa7mx1#

添加此文件

/etc/yum.repos.d/flume.repo

使用此内容

[flume]
Name=Packages for flume
baseurl=http://archive.cloudera.com/redhat/cdh/3/
enabled=1
gpgcheck=0

这将告诉您的yum命令查找安装flume ng的位置
当然,这很简单:

sudo yum install flume-ng
v64noz0r

v64noz0r2#

如果添加cloudera cdh apt存储库,则可以从软件包安装flume。看看这本指南。

zpgglvta

zpgglvta3#

ubuntu中的apache flume安装步骤:

Step 1 : Download the Latest Version of Flume. 

Step 2 : tar –xzvf apache-flume-1.5.0-bin.tar.gz

Step 3 : sudo mv apache-flume-1.5.2-bin /usr/local/flume

Step 4 : sudo nano ~/.bashrc

Step 5 : export FLUME_HOME=/usr/local/flume
export FLUME_CONF_DIR=$FLUME_HOME/conf
export FLUME_CLASS_PATH=$FLUME_CONF_DIR
export PATH=$FLUME_HOME/bin:$PATH

Step 6 : 

cp conf/flume-env.sh.template conf/flume-env.sh
Step 7 : 

sudo nano conf/flume-env.sh
JAVA_HOME=/usr/lib/jvm/jdk1.8.0
JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote"
Step 8: 
sudo nano flume-conf.properties.template 

agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity = 100

# Define a source on agent and connect to channel memoryChannel.

agent.sources.tail-source.type = exec
agent.sources.tail-source.command = tail -F /opt/hadoop-2.6.0/logs/hadoop-hadoop-datanode-node1.log
agent.sources.tail-source.channels = memoryChannel

# Define a sink that outputs to logger.

agent.sinks.log-sink.channel = memoryChannel
agent.sinks.log-sink.type = logger

agent.sinks.hdfs-sink.channel = memoryChannel
agent.sinks.hdfs-sink.type = hdfs
agent.sinks.hdfs-sink.hdfs.path = hdfs://node1:8020/flumedata/
agent.sinks.hdfs-sink.hdfs.fileType = DataStream

# Activate channel, source and sinks

agent.channels = memoryChannel
agent.sources = tail-source
agent.sinks = log-sink hdfs-sink
Start Flume-ng agent

source ~/.bashrc

flume-ng agent -n agent -f conf/flume.conf -Dflume.root.logger=DEBUG,console
flume-ng --help

希望这有帮助。

相关问题