我是新来的ApacheFlume。我创建了如下代理：
agent.sources=exec-source
agent.sinks=hdfs-sink
agent.channels=ch1
agent.sources.exec-source.type=exec
agent.sources.exec-source.command=tail -F /var/log/apache2/access.log
agent.sinks.hdfs-sink.type=hdfs
agent.sinks.hdfs-sink.hdfs.path=hdfs://<Host-Name of name node>/
agent.sinks.hdfs-sink.hdfs.filePrefix=apacheaccess
agent.channels.ch1.type=memory
agent.channels.ch1.capacity=1000
agent.sources.exec-source.channels=ch1
agent.sinks.hdfs-sink.channel=ch1我得到的结果是：
13/01/22 17:31:48 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 1
13/01/22 17:31:48 INFO node.FlumeNode: Flume node starting - agent
13/01/22 17:31:48 INFO properties.PropertiesFileConfigurationProvider: Configuration provider starting
13/01/22 17:31:48 INFO nodemanager.DefaultLogicalNodeManager: Node manager starting
13/01/22 17:31:48 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 9
13/01/22 17:31:48 INFO properties.PropertiesFileConfigurationProvider: Reloading configuration file:conf/flume_exec.conf
13/01/22 17:31:48 INFO conf.FlumeConfiguration: Added sinks: hdfs-sink Agent: agent
13/01/22 17:31:48 INFO conf.FlumeConfiguration: Processing:hdfs-sink
13/01/22 17:31:48 INFO conf.FlumeConfiguration: Processing:hdfs-sink
13/01/22 17:31:48 INFO conf.FlumeConfiguration: Processing:hdfs-sink
13/01/22 17:31:48 INFO conf.FlumeConfiguration: Processing:hdfs-sink
13/01/22 17:31:48 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent]
13/01/22 17:31:48 INFO properties.PropertiesFileConfigurationProvider: Creating channels
13/01/22 17:31:48 INFO properties.PropertiesFileConfigurationProvider: created channel ch1
13/01/22 17:31:48 INFO sink.DefaultSinkFactory: Creating instance of sink: hdfs-sink, type: hdfs
13/01/22 17:31:48 INFO hdfs.HDFSEventSink: Hadoop Security enabled: false
13/01/22 17:31:48 INFO nodemanager.DefaultLogicalNodeManager: Starting new configuration:{ sourceRunners:{exec-source=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:exec-source,state:IDLE} }} sinkRunners:{hdfs-sink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@715d44 counterGroup:{ name:null counters:{} } }} channels:{ch1=org.apache.flume.channel.MemoryChannel{name: ch1}} }
13/01/22 17:31:48 INFO nodemanager.DefaultLogicalNodeManager: Starting Channel ch1
13/01/22 17:31:48 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: CHANNEL, name: ch1, registered successfully.
13/01/22 17:31:48 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: ch1 started
13/01/22 17:31:48 INFO nodemanager.DefaultLogicalNodeManager: Starting Sink hdfs-sink
13/01/22 17:31:48 INFO nodemanager.DefaultLogicalNodeManager: Starting Source exec-source
13/01/22 17:31:48 INFO source.ExecSource: Exec source starting with command:tail -F /var/log/apache2/access.log
13/01/22 17:31:48 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: SINK, name: hdfs-sink, registered successfully.
13/01/22 17:31:48 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: hdfs-sink started但它不会将日志写入hdfs。
当我跑的时候 cat /var/log/apache2/access.log 而不是 tail –F /var/log/apache2/access.log 它运行，我的文件在hdfs上创建。
请帮帮我。
1条答案
按热度按时间qnzebej01#
默认情况下，“tail-f”只在开始时打印文件的最后10行。似乎10行不足以填充hdfs块，所以您看不到flume编写的任何内容。你可以：
尝试“tail-n$x-f”在开始时打印最后x行（x的值将根据hdfs设置中块的大小而变化）
等待access.log在flume运行时增长到足够大（同样，等待的时间取决于块的大小和access.log的增长率；在生产模式下，我认为它将足够快）
在flume.conf中添加以下行。它将强制flume每10秒滚动一次新文件，而不管写入数据的大小（假设它不是零）：
agent.sinks.hdfs-sink.hdfs.rollinterval=10
agent.sinks.hdfs\u sink.hdfs.rollsize=0