Describe the bug
We got an error:ERROR 1064 (HY000): Memory exceed limit. New partitioned Aggregation in spill Backend: xxx.xxx.xxx.xxx, fragment: xxx Used: 10528896, Limit: -1. You can change the limit by session variable exec_mem_limit.
incubator-doris/be/src/exec/new_partitioned_aggregation_node.cc
Lines 816 to 824 in 14c0881
| | Status NewPartitionedAggregationNode::Partition::Spill(bool more_aggregate_rows) { |
| | DCHECK(!parent->is_streaming_preagg_); |
| | DCHECK(!is_closed); |
| | DCHECK(!is_spilled()); |
| | // TODO(ml): enable spill |
| | std::stringstream msg; |
| | msg << "New partitioned Aggregation in spill"; |
| | LIMIT_EXCEEDED(parent->mem_tracker(), parent->state_, msg.str()); |
| | // RETURN_IF_ERROR(parent->state_->StartSpilling(parent->mem_tracker())); |
may caused by
incubator-doris/be/src/exec/new_partitioned_aggregation_node.cc
Lines 1131 to 1138 in 14c0881
| | bool got_memory; |
| | RETURN_IF_ERROR(partition->InitHashTable(&got_memory)); |
| | // Spill the partition if we cannot create a hash table for a merge aggregation. |
| | if (UNLIKELY(!got_memory)) { |
| | DCHECK(!is_streaming_preagg_) << "Preagg reserves enough memory for hash tables"; |
| | // If we're repartitioning, we will be writing aggregated rows first. |
| | RETURN_IF_ERROR(partition->Spill(level > 0)); |
| | } |
But whatever caused Spill() calling, if Spill() is disabled(), we shouldn't call it, right? And the error message is misleading.
Solutions
- Spill() return the right error msg "Spill is disabled. And the reason maybe mem limit... ". LIMIT_EXCEEDED just get one mem_tracker's limit, it's incorrect, we need to use lowest_limit().
PS. If Spill() is called coz Suballoctor->allocate() failed(got_memory==false), it may be not related to this plan's mem_trackers limit. - enable Spill(), it's unfamiliar to me, so we need more thorough discuss.
暂无答案!
目前还没有任何答案,快来回答吧!