具有内存通道的flume性能

hts6caw3  于 2021-05-29  发布在  Hadoop
关注(0)|答案(0)|浏览(283)

对于配置低于4gb的数据,需要3分钟。有没有办法减少时间。什么时候是最好的时机?我不认为有问题的hdfs接收器,但无论是源或通道。

  1. identify the components on agent memoryagent
  2. memoryagent.sources = tr_source
  3. memoryagent.sinks = tr_sink tr1_sink
  4. memoryagent.channels = tr_channel
  5. Configure the source
  6. memoryagent.sources.tr_source.type = spooldir
  7. memoryagent.sources.tr_source.spoolDir = /home/apps/flumetest
  8. memoryagent.sources.tr_source.deletePolicy = immediate
  9. memoryagent.sources.tr_source.batchSize = 100000
  10. memoryagent.sources.tr_source.deserializer.maxLineLength = 999999999
  11. memoryagent.sinks.tr_sink.type = hdfs
  12. memoryagent.sinks.tr_sink.hdfs.path = hdfspath/ymd=%Y%m%d
  13. memoryagent.sinks.tr_sink.hdfs.filePrefix = sink0
  14. memoryagent.sinks.tr_sink.hdfs.rollInterval = 0
  15. memoryagent.sinks.tr_sink.hdfs.rollCount = 0
  16. memoryagent.sinks.tr_sink.hdfs.rollSize = 1600000000
  17. memoryagent.sinks.tr_sink.hdfs.batchSize = 100000
  18. # memoryagent.sinks.tr_sink.hdfs.codeC = snappy
  19. memoryagent.sinks.tr_sink.hdfs.fileType = DataStream
  20. memoryagent.sinks.tr_sink.hdfs.writeFormat = Text
  21. memoryagent.sinks.tr_sink.hdfs.useLocalTimeStamp = true
  22. memoryagent.sinks.tr_sink.hdfs.callTimeout = 30000
  23. memoryagent.sinks.tr_sink.hdfs.threadsPoolSize = 20
  24. memoryagent.sinks.tr1_sink.type = hdfs
  25. memoryagent.sinks.tr1_sink.hdfs.path = hdfspath/ymd=%Y%m%d
  26. memoryagent.sinks.tr1_sink.hdfs.filePrefix = sink1
  27. memoryagent.sinks.tr1_sink.hdfs.rollInterval = 0
  28. memoryagent.sinks.tr1_sink.hdfs.rollCount = 0
  29. memoryagent.sinks.tr1_sink.hdfs.rollSize = 1600000000
  30. memoryagent.sinks.tr1_sink.hdfs.batchSize = 100000
  31. # memoryagent.sinks.tr1_sink.hdfs.codeC = snappy
  32. memoryagent.sinks.tr1_sink.hdfs.fileType = DataStream
  33. memoryagent.sinks.tr1_sink.hdfs.writeFormat = Text
  34. memoryagent.sinks.tr1_sink.hdfs.useLocalTimeStamp = true
  35. memoryagent.sinks.tr1_sink.hdfs.callTimeout = 30000
  36. memoryagent.sinks.tr1_sink.hdfs.threadsPoolSize = 20
  37. Configure a channel that buffers events in file
  38. memoryagent.channels.tr_channel.type = memory
  39. memoryagent.channels.tr_channel.capacity = 999999999
  40. memoryagent.channels.tr_channel.transactionCapacity = 99999999
  41. memoryagent.channels.tr_channel.type.byteCapacity = 161061273600
  42. Bind the source and sink to the channel
  43. memoryagent.sources.tr_source.channels = tr_channel
  44. memoryagent.sinks.tr1_sink.channel = tr_channel
  45. memoryagent.sinks.tr_sink.channel = tr_channel

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题