如何删除logstash conf文件中地理坐标中的引号(“”)

ilmyapht  于 2021-06-13  发布在  ElasticSearch
关注(0)|答案(1)|浏览(460)

我想在kibana的Map上表示一条线,Map应该是geojson结构。我拥有的数据是一组sql表,然后我将使用如下的logstash将它们传输到ElasticSearch

input{ ... }

filter{

 if [lat] and [lon] {
        mutate{convert =>  ["lat", "float"]}
        mutate{convert =>  ["lon", "float"]}
        mutate{convert =>  ["for_lat", "float"]}
        mutate{convert =>  ["for_lon", "float"]}

        mutate{
            add_field => {"[location-geotest][type]" => "multilinestring"}
            add_field => {"[location-geotest][coordinates]" => [["%{lon}", "%{lat}"]]}
            add_field => {"[location-geotest][coordinates]" => [["%{for_lon}", "%{for_lat}"]]}
            }
      }
}

但是,logstash conf文件未能索引elasticsearch上的数据

{
    :status=>400, 
    :action=>["index", {:_id=>"18022", :_index=>"geo_shape_test", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x687994b9>], 
    :response=> {
        "index"=>{
            "_index"=>"geo_shape_test", 
            "_type"=>"_doc", 
            "_id"=>"18022", 
            "status"=>400, 
            "error"=>{
                "type"=>"mapper_parsing_exception", 
                "reason"=>"failed to parse field [location-geotest] of type [geo_shape]", 
                "caused_by"=>{"type"=>"x_content_parse_exception", 
                    "reason"=>"[1:164] [geojson] failed to parse field [coordinates]", 
                    "caused_by"=>{
                        "type"=>"parse_exception", 
                        "reason"=>"geo coordinates must be numbers"
                    }
                }
            }
        }
    }
}

这是logstash试图索引的内容之一

{
                 "lat" => 37.567179953757886,
              "gps_id" => 10491,
           "timestamp" => 2020-11-22T06:10:45.000Z,
               "speed" => 17.25745240090587,
                 "lon" => 126.99598717854032,
             "for_lat" => 37.567179953757886,
          "@timestamp" => 2020-11-27T03:54:21.131Z,
             "for_lon" => 126.99598717854032,
            "@version" => "1",
    "location-geotest" => {
        "coordinates" => [
            [0] "[\"126.99598717854032\", \"37.567179953757886\"]",
            [1] "[\"126.99598717854032\", \"37.567179953757886\"]"
        ],
               "type" => "multilinestring"
    }
}

我想问题是。。。

"coordinates" => [
            [0] "[\"126.99598717854032\", \"37.567179953757886\"]",
            [1] "[\"126.99598717854032\", \"37.567179953757886\"]"
        ],

如果我换一个角色,它会是。。。

"coordinates" => [
            [0] [126.99598717854032, 37.567179953757886],
            [1] [126.99598717854032, 37.567179953757886]
        ],

但我找不到解决方法。

3xiyfsfu

3xiyfsfu1#

我认为问题是,正如你所说,坐标必须是浮点数而不是字符串。显然,mutate函数将值转换回字符串。如中所述
https://discuss.elastic.co/t/logstash-mutate-filter-always-stringifies-hash-and-array/25917
他们建议使用ruby脚本来代替。这是为linestring as完成的。
https://discuss.elastic.co/t/geo-shape-geo-link-problems-with-coordinates/179924/4
从提供的数据我不明白为什么你需要多行字符串?只有两个点,它应该足够存储为行字符串。
我试过了

filter{

 if [lat] and [lon] {

        mutate{
            convert =>  ["lat", "float"]
            convert =>  ["lon", "float"]
            convert =>  ["for_lat", "float"]
            convert =>  ["for_lon", "float"]

            add_field => {"[location-geotest][type]" => "linestring"}
        }

      ruby{
          code => "event.set('[location-geotest][coordinates]', [[event.get('lon'), event.get('lat')], [event.get('for_lon'), event.get('for_lat')]])"
      }
  }
}

得到结果:

"location-geotest" => {
           "type" => "linestring",
           "coordinates" => [
                 [0] [
                      [0] 126.99598717854032,
                      [1] 37.567179953757886
                     ],
                 [1] [
                      [0] 126.99598717854032,
                      [1] 37.567179953757886
                     ]
           ]
         }

索引正确。
如果您需要多个字符串,我想您需要更多的数据,并在ruby脚本中再添加一层数组。

相关问题