json—从单个输入生成多个jq输出文档，修改每个结果

euoag5mw 于 2021-06-14 发布在 ElasticSearch

关注(0)|答案(2)|浏览(534)

我想在我的json文件中做2个操作。我试着用jq和shell。
第一个：我想将parents元素转换为纯文本值
第二个：我想删除json树中的一个特定级别
输入：

{
  "template_first": {
    "order": 0,
    "index_patterns": [
      "first"
    ],
    "settings": {
      "index": {
        "codec": "best_compression",
        "refresh_interval": "30s",
        "analysis": {
          "normalizer": {
            "norm_case_insensitive": {
              "filter": "lowercase",
              "type": "custom"
            }
          }
        },
        "number_of_shards": "1",
        "number_of_replicas": "1"
      }
    },
    "mappings": {
      "_doc": {
        "dynamic": true,
        "dynamic_templates": [
          {
            "strings": {
              "mapping": {
                "type": "keyword"
              },
              "match_mapping_type": "string"
            }
          }
        ],
        "properties": {
          "log.id": {
            "type": "keyword"
          },
          "host.indexer.hostname": {
            "type": "keyword"
          },
          "ts_indexer": {
            "format": "strict_date_optional_time||epoch_millis",
            "type": "date"
          }
        }
      }
    }
  },
  "template_second": {
    "order": 0,
    "index_patterns": [
      "second"
    ],
    "settings": {
      "index": {
        "codec": "best_compression",
        "refresh_interval": "30s",
        "analysis": {
          "normalizer": {
            "norm_case_insensitive": {
              "filter": "lowercase",
              "type": "custom"
            }
          }
        },
        "number_of_shards": "1",
        "number_of_replicas": "1"
      }
    },
    "mappings": {
      "_doc": {
        "dynamic": true,
        "dynamic_templates": [
          {
            "strings": {
              "mapping": {
                "type": "keyword"
              },
              "match_mapping_type": "string"
            }
          }
        ],
        "properties": {
          "log.id": {
            "type": "keyword"
          },
          "host.indexer.hostname": {
            "type": "keyword"
          },
          "ts_indexer": {
            "format": "strict_date_optional_time||epoch_millis",
            "type": "date"
          }
        }
      }
    }
  }
}

文件中有两个json对象

{
    "template_first" : { ...},
    "template_second" : { ... }
     }

第一个修改来自此命令的外观
放置模板/模板编号
而不是第一个json对象的键。
所以预期的结果

PUT _template/template_first
  {...}
PUT _template/template_second
  {...}

第二个变化是删除了文档级别
之前：

"mappings": {
    "_doc": {
      "dynamic": true,
      "dynamic_templates": [
        {
          "strings": {
            "mapping": {
              "type": "keyword"
            },
            "match_mapping_type": "string"
          }
        }
      ],
      "properties": {
        "log.id": {
          "type": "keyword"
        },
        "host.indexer.hostname": {
          "type": "keyword"
        },
        "ts_indexer": {
          "format": "strict_date_optional_time||epoch_millis",
          "type": "date"
        }
      }
    }
  }

预期结果

"mappings": {
    "dynamic": true,
    "dynamic_templates": [
      {
        "strings": {
          "mapping": {
            "type": "keyword"
          },
          "match_mapping_type": "string"
        }
      }
    ],
    "properties": {
      "log.id": {
        "type": "keyword"
      },
      "host.indexer.hostname": {
        "type": "keyword"
      },
      "ts_indexer": {
        "format": "strict_date_optional_time||epoch_millis",
        "type": "date"
      }
    }
  }

所以实际结果是这样的

PUT _template/template_first
  {
  "order": 0,
  "index_patterns": [
    "first"
  ],
  "settings": {
    "index": {
      "codec": "best_compression",
      "refresh_interval": "30s",
      "analysis": {
        "normalizer": {
          "norm_case_insensitive": {
            "filter": "lowercase",
            "type": "custom"
          }
        }
      },
      "number_of_shards": "1",
      "number_of_replicas": "1"
    }
  },
  "mappings": {
    "dynamic": true,
    "dynamic_templates": [
      {
        "strings": {
          "mapping": {
            "type": "keyword"
          },
          "match_mapping_type": "string"
        }
      }
    ],
    "properties": {
      "log.id": {
        "type": "keyword"
      },
      "host.indexer.hostname": {
        "type": "keyword"
      },
      "ts_indexer": {
        "format": "strict_date_optional_time||epoch_millis",
        "type": "date"
      }
    }
  }
}
PUT _template/template_second
  {
  "order": 0,
  "index_patterns": [
    "second"
  ],
  "settings": {
    "index": {
      "codec": "best_compression",
      "refresh_interval": "30s",
      "analysis": {
        "normalizer": {
          "norm_case_insensitive": {
            "filter": "lowercase",
            "type": "custom"
          }
        }
      },
      "number_of_shards": "1",
      "number_of_replicas": "1"
    }
  },
  "mappings": {
    "dynamic": true,
    "dynamic_templates": [
      {
        "strings": {
          "mapping": {
            "type": "keyword"
          },
          "match_mapping_type": "string"
        }
      }
    ],
    "properties": {
      "log.id": {
        "type": "keyword"
      },
      "host.indexer.hostname": {
        "type": "keyword"
      },
      "ts_indexer": {
        "format": "strict_date_optional_time||epoch_millis",
        "type": "date"
      }
    }
  }
}

我实现了第二个更改：使用以下命令删除json数组的一个级别

jq  'keys[] as $k | map( .mappings =.mappings._doc   )' template.json

但我不知道如何在同一时间做第一次改变和第二次改变。
我试着这样循环到数组中，但没有成功

for row in $(jq 'keys[] as $k | "\($k)"' template.json); do
    _jq() {
     echo ${row} 
    }
   echo $(_jq '.name')
done

shell elasticsearch JSON loops jq

来源：https://stackoverflow.com/questions/64431085/generate-multiple-jq-output-documents-from-a-single-input-modifying-each-result

2条答案

按热度按时间

vxf3dgd41#

我在工作上遇到了一些麻烦 done < <( 这导致我的shell出现语法错误（不知道为什么）。所以我修改了你的剧本如下：

jq -j 'to_entries[] | .key as $template_name | .value as $template_content | ($template_name, "\u0000", ($template_content | (.mappings = .mappings._doc) | tojson), "\u0000")' < infile.json |
while IFS= read -r -d '' template_name && IFS= read -r -d '' template_content; do
  echo "PUT _template/$template_name"
  printf '%s\n' "$template_content"
done

做得很好！谢谢查尔斯

赞(0）回复(0）举报 2021-06-14

xkrw2x1b2#

打电话 jq 只需一次，让它编写一个以nul分隔的模板名称/修改后的模板内容对列表（bash while read 循环然后可以迭代）：

while IFS= read -r -d '' template_name && IFS= read -r -d '' template_content; do
  echo "We want to do PUT the following to _template/$template_name"
  printf '%s\n' "$template_content"
done < <(
  jq -j '
    to_entries[] |
    .key as $template_name |
    .value as $template_content |
    ($template_name, "\u0000",
     ($template_content | (.mappings = .mappings._doc) | tojson), "\u0000")
  ' <infile.json
)

赞(0）回复(0）举报 2021-06-14

我来回答

json—从单个输入生成多个jq输出文档，修改每个结果

2条答案

相关问题

热门标签

最新问答