告警延迟分析

ryoqjall  于 2022-10-19  发布在  其他
关注(0)|答案(3)|浏览(147)

背景

微服务>50,告警>2w时,经常出现告警延迟,最多时候达到1h以上

原因分析

SendExcutor每5min拉取一次告警,通过arthas分析,主要是在 m_decoratorManager.generateTitleAndContent 耗时长,发现是其中有一条SQL执行很慢
SELECT al.id,al.domain,al.alert_time,al.category,al. type ,al.content,al.metric,al.creation_date FROM alert al WHERE al.alert_time >= ? AND al.alert_time <= ? AND al.category = ? AND al.domain = ? ORDER BY al.alert_time asc

解决策略

增加索引index idx_alert_time_category_domain( alert_time , category , domain );
alter table alert add index idx_alert_time_category_domain( alert_time , category , domain );

建议

在初始化的CatApplication.sql中,补充该索引

8gsdolmq

8gsdolmq1#

麻烦顺便发个pull request,谢谢!…

On Aug 15, 2022, at 14:48, romanticmj***@***.***> wrote: 背景 微服务>50,告警>2w时,经常出现告警延迟,最多时候达到1h以上 原因分析 SendExcutor每5min拉取一次告警,通过arthas分析,主要是在 m_decoratorManager.generateTitleAndContent 耗时长,发现是其中有一条SQL执行很慢 SELECT al.id,al.domain,al.alert_time,al.category,al.type,al.content,al.metric,al.creation_date FROM alert al WHERE al.alert_time >= ? AND al.alert_time <= ? AND al.category = ? AND al.domain = ? ORDER BY al.alert_time asc 解决策略 增加索引index idx_alert_time_category_domain(alert_time,category,domain); alter table alert add index idx_alert_time_category_domain(alert_time,category,domain); 建议 在初始化的CatApplication.sql中,补充该索引 — Reply to this email directly, view it on GitHub <#2243>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AASQE73KM2TVQSE5SMT6QRDVZHR2BANCNFSM56RIPGQQ >. You are receiving this because you are subscribed to this thread.

1zmg4dgp

1zmg4dgp3#

Merged into master…

On Aug 17, 2022, at 10:31, romanticmj***@***.******@***.***>> wrote: PR: #2245 < #2245 > — Reply to this email directly, view it on GitHub <#2243 (comment)>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AASQE77BRS2M5PA6YTRIE2TVZRFJ3ANCNFSM56RIPGQQ >. You are receiving this because you commented.

相关问题