stream日志文件从windows到linux中的hdfs

lsmepo6l  于 2021-06-04  发布在  Flume
关注(0)|答案(1)|浏览(422)

在linux中如何将日志文件从Windows7流式传输到hdfs?
windows中的Flume出现错误
我已在windows 7(节点1)上安装“flume-node-0.9.3”。“flumenode”服务正在运行并且localhost:35862 is 无障碍。
在windows中,日志文件位于“c:/logs/weblogic.log”
centos linux(节点2)中的flume代理也在运行。
在windows计算机中,java\ U home变量设置为“c:\program files\java\jre7”
java.exe文件位于“c:\program files\java\jre7\bin\java.exe”
flume节点安装在“c:\program files\cloudera\flume 0.9.3”
以下是flume-src.conf文件,位于Windows7上flume的“conf”文件夹中(节点1)

  1. source_agent.sources = weblogic_server
  2. source_agent.sources.weblogic_server.type = exec
  3. source_agent.sources.weblogic_server.command = tail -f C:/logs/Weblogic.log
  4. source_agent.sources.weblogic_server.batchSize = 1
  5. source_agent.sources.weblogic_server.channels = memoryChannel
  6. source_agent.sources.weblogic_server.interceptors = itime ihost itype
  7. source_agent.sources.weblogic_server.interceptors.itime.type = timestamp
  8. source_agent.sources.weblogic_server.interceptors.ihost.type = host
  9. source_agent.sources.weblogic_server.interceptors.ihost.useIP = false
  10. source_agent.sources.weblogic_server.interceptors.ihost.hostHeader = host
  11. source_agent.sources.weblogic_server.interceptors.itype.type = static
  12. source_agent.sources.weblogic_server.interceptors.itype.key = log_type
  13. source_agent.sources.weblogic_server.interceptors.itype.value = apache_access_combined
  14. source_agent.channels = memoryChannel
  15. source_agent.channels.memoryChannel.type = memory
  16. source_agent.channels.memoryChannel.capacity = 100
  17. source_agent.sinks = avro_sink
  18. source_agent.sinks.avro_sink.type = avro
  19. source_agent.sinks.avro_sink.channel = memoryChannel
  20. source_agent.sinks.avro_sink.hostname = 10.10.201.40
  21. source_agent.sinks.avro_sink.port = 41414

我试图通过在flume文件夹中执行以下命令来运行上述文件:

  1. C:\Program Files\Cloudera\Flume 0.9.3>"C:\Program Files\Java\jre7\bin\java.exe"
  2. -Xmx20m -Dlog4j.configuration=file:///%CD%\conf\log4j.properties -cp "C:\Program Files\Cloudera\Flume 0.9.3\lib*" org.apache.flume.node.Application
  3. -f C:\Program Files\Cloudera\Flume 0.9.3\conf\flume-src.conf -n source_agent

但它给出了以下信息:

  1. Error: Could not find or load main class Files\Cloudera\Flume

下面是运行在centos(节点2)中的trg-node.conf文件。centos节点工作正常:

  1. collector.sources = AvroIn
  2. collector.sources.AvroIn.type = avro
  3. collector.sources.AvroIn.bind = 0.0.0.0
  4. collector.sources.AvroIn.port = 41414
  5. collector.sources.AvroIn.channels = mc1 mc2
  6. collector.channels = mc1 mc2
  7. collector.channels.mc1.type = memory
  8. collector.channels.mc1.capacity = 100
  9. collector.channels.mc2.type = memory
  10. collector.channels.mc2.capacity = 100
  11. collector.sinks = HadoopOut
  12. collector.sinks.HadoopOut.type = hdfs
  13. collector.sinks.HadoopOut.channel = mc2
  14. collector.sinks.HadoopOut.hdfs.path =/user/root
  15. collector.sinks.HadoopOut.hdfs.callTimeout = 150000
  16. collector.sinks.HadoopOut.hdfs.fileType = DataStream
  17. collector.sinks.HadoopOut.hdfs.writeFormat = Text
  18. collector.sinks.HadoopOut.hdfs.rollSize = 0
  19. collector.sinks.HadoopOut.hdfs.rollCount = 10000
  20. collector.sinks.HadoopOut.hdfs.rollInterval = 600
ego6inou

ego6inou1#

问题是由于此路径中的程序和文件之间存在空白:
c:程序文件\cloudera\flume 0.9.3
考虑在没有空格的路径上安装Flume,它会像一个魅力。

相关问题