这是我的 FlumeHadoop.conf
文件。
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = spooldir
a1.sources.r1.channels = c1
a1.sources.r1.spoolDir = /home/rabindra/idirectory
a1.sources.r1.basenameHeader=true
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 1000
a1.channels.c1.byteCapacityBufferPercentage = 20
a1.channels.c1.byteCapacity = 131072000
a1.sinks.k1.channel = c1
# properties of k1-sink
a1.sinks.k1.type = hdfs
# a1.sinks.k1.hdfs.path = hdfs://namenode/flumesource/source1
a1.sinks.k1.hdfs.path = hdfs://localhost/logdata
a1.sinks.k1.hdfs.filePrefix=%{basename}
a1.sinks.k1.hdfs.fileSuffix=.txt
a1.sinks.k1.rollInterval=0
a1.sinks.k1.hdfs.deletePolicy=immediate
a1.sinks.k1.hdfs.rollSize=131072000
a1.sinks.k1.hdfs.rollCount=0
a1.sinks.k1.hdfs.idleTimeout=0
a1.sinks.k1.hdfs.maxOpenFiles = 10000
在远程主机中,比如192.168.7.43,有一个目录rdirectory和目录path/home/alex/rdirectory。
我要将远程文件夹rdirectory指向损坏目录。怎么可能?我正在努力改变
a1.sources.r1.spoolDir = alex@192.168.7.43:/home/alex/rdirectory
但这会产生异常java.lang.illegalstateexception:目录不存在:/opt/flume/alex@192.168.7.43:/home/alex/r目录。
2条答案
按热度按时间lvmkulzt1#
你试过运行这个选项吗
在192.168.7.4 spool dir[source]-->netcat[sink]中运行flume代理
在本地计算机netcat中运行flume代理[source]-->hdfs sink
v1uwarro2#
1.在主机192.168.7.43上安装samba,config folder/home/alex/rdirectory可以间隔//192.168.7.43/rdirectory
2.在flume代理主机上挂载共享目录,然后可以像本地目录一样使用挂载目录。
下面是192.168.59.166上装载目录到主机h0045170的示例