用于Logstash的Serilog HTTP接收器自定义格式

prdp8dxp  于 2022-12-09  发布在  Logstash
关注(0)|答案(2)|浏览(208)

我正在使用Serilog HTTP sink在我的.Net核心项目中记录到Logstash。在startup.cs中,我有以下代码来启用serilog。

Log.Logger = new LoggerConfiguration()
        .Enrich.FromLogContext()
        .WriteTo.Http("http://mylogstashhost.com:5000").Enrich.WithProperty("user", "xxx").Enrich.WithProperty("serviceName", "yyy")
        .MinimumLevel.Warning()
        .CreateLogger();

这段代码将日志发送到给定的http地址。我可以在fiddler上看到下面的json被发布到logstash,logstash返回“ok”消息。

{"events":[{"Timestamp":"2018-10-19T18:16:27.6561159+01:00","Level":"Warning","MessageTemplate":"abc","RenderedMessage":"abc","user":"xxx","serviceName":"yyy","Properties":{"ActionId":"b313b8ed-0baf-4d75-a6e2-f0dbcb941f67","ActionName":"MyProject.Controllers.HomeController.Index","RequestId":"0HLHLQMV1EBCJ:00000003","RequestPath":"/"}}]}

但是当我检查Kibana时,我看不到这个日志。我试着找出是什么原因导致的,我意识到如果我以下面的格式发送JSON,我就可以看到日志。

{"Timestamp":"2018-10-19T18:16:27.6561159+01:00","Level":"Warning","MessageTemplate":"abc","RenderedMessage":"abc","user":"xxx","serviceName":"yyy","Properties":{"ActionId":"b313b8ed-0baf-4d75-a6e2-f0dbcb941f67","ActionName":"MyProject.Controllers.HomeController.Index" ,"RequestId":"0HLHLQMV1EBCJ:00000003","RequestPath":"/"}}

所以Logstash不想让事件出现在Events{}中,而且它还想让“user”和“ServiceName”标记离开“Properties”。有没有办法像这样格式化我的Json?

ckx4rj1h

ckx4rj1h1#

经过一些研究和帮助,基本上实现自定义格式,应该实现像ITextFormatter,BatchFormatter等接口。
我可以通过稍微修改ArrayBatchFormatter来实现我需要的格式:

public class MyFormat : BatchFormatter
{
    /// <summary>
    /// Initializes a new instance of the <see cref="ArrayBatchFormatter"/> class.
    /// </summary>
    /// <param name="eventBodyLimitBytes">
    /// The maximum size, in bytes, that the JSON representation of an event may take before it
    /// is dropped rather than being sent to the server. Specify null for no limit. Default
    /// value is 256 KB.
    /// </param>
    public MyFormat(long? eventBodyLimitBytes = 256 * 1024): base(eventBodyLimitBytes)
    {

    }

    /// <summary>
    /// Format the log events into a payload.
    /// </summary>
    /// <param name="logEvents">
    /// The events to format.
    /// </param>
    /// <param name="output">
    /// The payload to send over the network.
    /// </param>
    public override void Format(IEnumerable<string> logEvents, TextWriter output)
    {
        if (logEvents == null) throw new ArgumentNullException(nameof(logEvents));
        if (output == null) throw new ArgumentNullException(nameof(output));

        // Abort if sequence of log events is empty
        if (!logEvents.Any())
        {
            return;
        }

        output.Write("[");

        var delimStart = string.Empty;

        foreach (var logEvent in logEvents)
        {
            if (string.IsNullOrWhiteSpace(logEvent))
            {
                continue;
            }
            int index = logEvent.IndexOf("{");

            string adjustedString = "{\"user\":\"xxx\",\"serviceName\" : \"yyy\"," + logEvent.Substring(1);
            if (CheckEventBodySize(adjustedString))
            {
                output.Write(delimStart);
                output.Write(adjustedString);
                delimStart = ",";
            }
        }

        output.Write("]");
    }
}
3bygqnnd

3bygqnnd2#

我想用这个变体来扩展@nooaa answer。我建议使用Newtonsoft.Json.Linq,而不是操作字符串来添加新对象。这样你就可以追加、添加或删除对象本身的现有属性。
此外,您可以合并事件的所有输出,并在最后执行一次output.write,而不是在每个事件之后执行output.write(性能有点问题)

public override void Format(IEnumerable<string> logEvents, TextWriter output)
{
    if (logEvents == null) throw new ArgumentNullException(nameof(logEvents));
    if (output == null) throw new ArgumentNullException(nameof(output));

    // Abort if sequence of log events is empty
    if (!logEvents.Any())
    {
        return;
    }
    
    List<object> updatedEvents = new List<object>();
    foreach (string logEvent in logEvents)
    {
        if (string.IsNullOrWhiteSpace(logEvent))
        {
            continue;
        }

        // Parse the log event
        var obj = JObject.Parse(logEvent);

        // Add New entries 
        obj["@source_host"] = obj["fields"]["MachineName"].Value<string>().ToLower();

        // Remove any entries you are not interested in
        ((JObject)obj["fields"]).Remove("MachineName");
                
        // Default tags for any log that goes out of your app.
        obj["@tags"] = new JArray() { "appName", "api" };

        // Additional tags from end points (custom based on routes)
        if (obj["fields"]["tags"] != null) 
        {
            ((JArray)obj["@tags"]).Merge((JArray)obj["fields"]["tags"]);
            ((JObject)obj["fields"]).Remove("tags");
        }
       
        updatedEvents.Add(obj);
    }

    output.Write(JsonConvert.SerializeObject(updatedEvents));
}

更新Release Notes v8.0.0

1.在最新版本中,您不再覆盖该方法。

namespace Serilog.Sinks.Http.BatchFormatters {
  public class MyCustomFormatter: IBatchFormatter {
    public void Format(IEnumerable<string> logEvents, TextWriter output) {
      ...
    }
  }
}

1.你也不提供任何构造器。
1.将队列限制字节与batchFormattertextFormatter相加

WriteTo.Http(new Uri(), 
  batchFormatter: new MyCustomFormatter(),
  queueLimitBytes: 50 * ByteSize.MB,
  textFormatter: new ElasticsearchJsonFormatter());

相关问题