我试图从alb日志中获得每天请求url的最大点击量(吞吐量)。我使用表投影来划分表。尝试找出查询,以获得过去1-3年中所有URL每分钟最大点击量的所需结果。结果应该是这样的(只是一个示例,时间戳可以是任何格式)
timestampurlmax点击率每分钟12-29-2019 8:01 amurl11072012-29-2019 10:35 amurl22132912-29-2019 10:35 amurl33742012-30-2019 11:53 amurl1589812-30-2019 01:30 pmurl21423012-30-2019 05:19 pmurl320000
表创建查询:
CREATE EXTERNAL TABLE IF NOT EXISTS alb_logs (
type string,
time string,
elb string,
client_ip string,
client_port int,
target_ip string,
target_port int,
request_processing_time double,
target_processing_time double,
response_processing_time double,
elb_status_code string,
target_status_code string,
received_bytes bigint,
sent_bytes bigint,
request_verb string,
request_url string,
request_proto string,
user_agent string,
ssl_cipher string,
ssl_protocol string,
target_group_arn string,
trace_id string,
domain_name string,
chosen_cert_arn string,
matched_rule_priority string,
request_creation_time string,
actions_executed string,
redirect_url string,
lambda_error_reason string,
target_port_list string,
target_status_code_list string,
classification string,
classification_reason string
)
PARTITIONED BY ( `partition_date` string)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
'serialization.format' = '1',
'input.regex' =
'([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*):([0-9]*) ([^ ]*)[:-]([0-9]*) ([-.0-9]*) ([-.0-9]*) ([-.0-9]*) (|[-0-9]*) (-|[-0-9]*) ([-0-9]*) ([-0-9]*) \"([^ ]*) ([^ ]*) (- |[^ ]*)\" \"([^\"]*)\" ([A-Z0-9-]+) ([A-Za-z0-9.-]*) ([^ ]*) \"([^\"]*)\" \"([^\"]*)\" \"([^\"]*)\" ([-.0-9]*) ([^ ]*) \"([^\"]*)\" \"([^\"]*)\" \"([^ ]*)\" \"([^\s]+?)\" \"([^\s]+)\" \"([^ ]*)\" \"([^ ]*)\"')
LOCATION 's3://your-alb-logs-directory/AWSLogs/<ACCOUNT-ID>/elasticloadbalancing/<REGION>/';
TBLPROPERTIES ('projection.enabled'='true',
'projection.partition_date.format'='yyyy/MM/dd',
'projection.partition_date.interval'='1',
'projection.partition_date.interval.unit'='DAYS',
'projection.partition_date.range'='2018/01/01,NOW',
'projection.partition_date.type'='date',
'storage.location.template'='s3://your-alb-logs-directory/AWSLogs/<ACCOUNT-ID>/elasticloadbalancing/<REGION>/${partition_date}')
1条答案
按热度按时间rsaldnfx1#
您可以尝试:
说明:公共表表达式(
cte
)将计算每个url每分钟的点击次数,然后从中提取达到最大点击次数的分钟数(使用max_by
函数)和最大点击数,按day
以及url
.参见文档:
max_by
功能:https://prestodb.io/docs/current/functions/aggregate.html#id2date_trunc
功能:https://prestodb.io/docs/current/functions/datetime.html