Fluentd可以将日志发送到Logstash吗?

ie3xauqp  于 12个月前  发布在  Logstash
关注(0)|答案(4)|浏览(234)

我一整天都在尝试这样做。我想通过fluentd日志引擎将日志从Docker发送到FluentD,然后从fluentd将这些日志发送到logstash进行处理。
我一直从logstash得到这个错误:

{:timestamp=>"2016-03-09T23:29:19.388000+0000",
 :message=>"An error occurred. Closing connection",
 :client=>"172.18.0.1:57259", :exception=>#<TypeError: can't convert String into Integer>,
 :backtrace=>["org/jruby/RubyTime.java:1073:in `at'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-event-2.2.2-java/lib/logstash/timestamp.rb:27:in `at'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-fluent-2.0.2-java/lib/logstash/codecs/fluent.rb:41:in `decode'", 
"org/msgpack/jruby/MessagePackLibrary.java:195:in `each'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-fluent-2.0.2-java/lib/logstash/codecs/fluent.rb:40:in `decode'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-3.0.2/lib/logstash/inputs/tcp.rb:153:in `handle_socket'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-3.0.2/lib/logstash/inputs/tcp.rb:143:in `server_connection_thread'"], :level=>:error}

字符串
基本的logstash配置:

input {
  tcp {
    port => 4000
    codec => "fluent"
  }
}

output {
  stdout {
  }
}


基本的fluentd配置:

<source>
  @type forward
</source>

<match docker.json>
  @type forward
  send_timeout 60s 
  recover_wait 10s 
  heartbeat_type none
  phi_threshold 16
  hard_timeout 60s 

  <server>
    name logstash
    host 172.18.0.2
    port 4000
    weight 60
  </server>
</match>

<match docker.**>
  @type stdout
</match>


有人会认为这会起作用,但我已经发现Logstash不会:

  • 使用fluentd的forward_out心跳配置。
  • Logstash不会打开与TCP相同端口的UDP端口。
  • 上面的错误。

如果我在Ruby中创建Fluentd消息包消息并手动发送,则上述配置确实有效。但关键是我希望Fluentd在本地管理日志并将其发送到外部logstash服务器,以便将消息正确处理为JSON。

baubqpgj

baubqpgj1#

我们找到了一种让fluent -> logstash工作的方法。设置time_as_integer true。fluentd端的最小配置是

<source>
  @type http
  @id input_http
  port 8888
</source>

<match **>
  @type forward
  time_as_integer true
  <server>
    host localhost
    port 24114
  </server>
</match>

字符串
它在https://docs.fluentd.org/v0.12/articles/in_forward#i-got-messagepackunknownexttypeerror-error-why中被提到是非常隐藏的。在logstash方面,使用最新的版本(6.2.4),然后简单地配置fluent codec,tcp输入如下:

input {
  tcp {
    codec => fluent
    port => 24114
  }
}

filter {
}

output {
  stdout { codec => rubydebug }
}


试验

curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test


使用time_as_integer设置,logstash输出将看起来很好,如。

{
          "port" => 32844,
      "@version" => "1",
          "host" => "localhost",
          "json" => "message",
    "@timestamp" => 2018-04-26T15:14:28.000Z,
          "tags" => [
        [0] "debug.test"
    ]
}


如果没有它,

[2018-04-26T15:16:00,115][ERROR][logstash.codecs.fluent   ] Fluent parse error, original data now in message field {:error=>#<MessagePack::UnknownExtTypeError: unexpected extension type>, :data=>["fluent.info", "\x92\xD7\u0000Z\xE1\xEC\xF4\u0006$\x96傦worker\u0000\xA7message\xD9&fluentd worker is now running worker=0", {"size"=>1, "compressed"=>"text"}]}
{
          "port" => 32972,
      "@version" => "1",
       "message" => [
        [0] "fluent.info",
        [1] "\x92\xD7\u0000Z\xE1\xEC\xF4\u0006$\x96傦worker\u0000\xA7message\xD9&fluentd worker is now running worker=0",
        [2] {
                  "size" => 1,
            "compressed" => "text"
        }
    ],
          "host" => "localhost",
    "@timestamp" => 2018-04-26T15:16:00.116Z,
          "tags" => [
        [0] "_fluentparsefailure"
    ]
}

vof42yt1

vof42yt12#

首先,没有办法将数据从Fluentd传输到Logstash。我们需要编写任何Fluentd输出插件来将数据发送到Logstash,或者编写任何Logstash输入插件来从Fluentd接收数据。
仅供参考:有一些插件用于Logstash -> Fluentd的方向:

  • fluent-plugin-beats(用于Elastic beats协议的fluentd输入插件)
  • logstash-output-fluentd(logstash输出插件,用于将数据发送到Fluentd)
ve7v8dk2

ve7v8dk23#

你可以直接转发到logstash tcp输入。
这个open-source flunetd output plugin将以json格式(也支持ssl/tls)直接将数据发送到logstash tcp输入(或任何其他接收器)。
第一次看到这个question

s2j5cfk0

s2j5cfk04#

您可以在logstash中使用http input,在fluentd中使用http output
logstash.conf

input {
  http {
   additional_codecs => {}
   codec => json { target => "[document]" }
   port => XXXX
   tags => ["fluentd"]
  }
}

字符串
fluentd.conf

<match **>
    @type http
    @id id
    content_type application/json
    endpoint http://server:XXXX
    json_array true
</match>


下载Fluentd-1.15.3
logstash-8.10.4-1

相关问题