kubernetes 当动态源ID更改时,Datadog会继续创建监视器

ldxq2e6h  于 2022-11-28  发布在  Kubernetes
关注(0)|答案(1)|浏览(117)

我有一个由terraform生成的Datadog监视器。主要查询如下:

sum(last_1m):avg:app.application.health{application.health:healthy,cluster_name:${local.eks_cluster_name},!source:api-service-full} by {source}.as_count() < 60"

问题在于系统重启后,{source}容器更改了其名称。
例如,从app-tier-1-1abc-agent
到应用程序层1-定义2-代理
Datadog不会更新或删除旧监视器,而是创建新监视器,并使旧监视器处于警报和N/A状态。
还有什么可以改进的吗?所有的想法都很感谢,谢谢!

rsaldnfx

rsaldnfx1#

通过发送API调用来编辑关闭和启动时的每个监视器查询,解决了此问题。
我做了一个非常笨拙的bash脚本,因为我找不到一种方法来在bash的变量中存储curl的数据,但是如果使用其他脚本语言,这可以用更少的代码完成,例如用于2个监视器

monitor_id=$(curl -L -X GET "https://api.datadoghq.com/api/v1/monitor" \
                  -H "Content-Type: application/json" \
                  -H "DD-API-KEY: ${DATADOG_API_KEY}" \
                  -H "DD-APPLICATION-KEY: ${DATADOG_APP_KEY}" \
                  --data-raw '{"tags":["application_id:'$APP_ID_LOWERCASE'"]}'| jq -r ' .[] | select((.name |endswith("XXXXXXX Heartbeats") or endswith("XXXXX HeartBeat monitoring")) and (.tags[]=="application_id:'$APP_ID_LOWERCASE'")) | .id')
        
      # curl gets cluster name used in queries
      CLUSTER_NAME=$(curl -L -X GET "https://api.datadoghq.com/api/v1/monitor" \
                  -H "Content-Type: application/json" \
                  -H "DD-API-KEY: ${DATADOG_API_KEY}" \
                  -H "DD-APPLICATION-KEY: ${DATADOG_APP_KEY}" \
                  --data-raw '{"tags":["application_id:'$APP_ID_LOWERCASE'"]}'| jq -r ' .[] | select((.name |endswith("XXXX HeartBeat monitoring")) and (.tags[]=="application_id:'$APP_ID_LOWERCASE'")) | .query' | awk -F',cluster_name:|,' '{print $2}')  
      
      # For each monitor id edit monitor query
      while IFS= read -r monitors 
        do    
          # curl gets monitor name
          monitor_name=$(curl -L -X GET "https://api.datadoghq.com/api/v1/monitor/"$monitors"" \
                          -H "Content-Type: application/json" \
                          -H "Accept: application/json" \
                          -H "DD-API-KEY: ${DATADOG_API_KEY}" \
                          -H "DD-APPLICATION-KEY: ${DATADOG_APP_KEY}" | jq -r .name)
           
            # checking which monitor query to send based on name, the queries are hardcoded because I couldn't find a way to set the query as a variable in bash
            if [[ $monitor_name == *"XXXXXX HeartBeat monitoring"* ]]; then 
                shutdown_query=$(curl -L -X PUT "https://api.datadoghq.com/api/v1/monitor/"$monitors"" \
                      -H "Accept: application/json" \
                      -H "Content-Type: application/json" \
                      -H "DD-API-KEY: ${DATADOG_API_KEY}" \
                      -H "DD-APPLICATION-KEY: ${DATADOG_APP_KEY}" \
                      --data-raw '{"query":"sum(last_1m):avg:health{application.health:healthy,cluster_name:'${CLUSTER_NAME}',!source:XXXXXX-full-1,source:DUMMY-VALUE-TO-RESEt-QUERY} by {source}.as_count() < 60"}')
            elif
              [[ $monitor_name == *"XXXXXX Instance Heartbeats"* ]]; then 
                shutdown_query=$(curl -L -X PUT "https://api.datadoghq.com/api/v1/monitor/"$monitors"" \
                      -H "Accept: application/json" \
                      -H "Content-Type: application/json" \
                      -H "DD-API-KEY: ${DATADOG_API_KEY}" \
                      -H "DD-APPLICATION-KEY: ${DATADOG_APP_KEY}" \
                      --data-raw '{"query":"sum(last_1m):avg:heartbeat{cluster_name:'${CLUSTER_NAME}',source:DUMMY-VALUE-TO-RESEt-QUERY} by {source}.as_count() < 3"}') 
            fi
        done <<< "$monitor_id"

然后,只需删除启动时的虚拟查询值,它将拿起您的新显示器,而忘记不存在的

相关问题