Paddle 预测部署的时候, qps高的时候报错

kh212irz  于 4个月前  发布在  其他
关注(0)|答案(5)|浏览(46)
  • 版本、环境信息:
       1)PaddlePaddle版本:2.2.2
       2)CPU:Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
       4)系统环境:CentOS release 4.3 (Final)

预测的时候, 低qps的时候请求没有问题, qps高起来会报下述错误, 请问原因有可能是什么呀:

C++ Traceback (most recent call last):

0 paddle::AnalysisPredictor::ZeroCopyRun()
1 paddle::framework::NaiveExecutor::Run()
2 paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, paddle::platform::Place const&)
3 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&) const
4 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&, paddle::framework::RuntimeContext*) const
5 std::_Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CPUPlace, false, 0ul, paddle::operators::RNNCPUKernel<paddle::platform::CPUDeviceContext, float>, paddle::operators::RNNCPUKernel<paddle::platform::CPUDeviceContext, double> >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&) #1 }>::_M_invoke(std::_Any_data const&, paddle::framework::ExecutionContext const&)
6 paddle::operators::RNNCPUKernel<paddle::platform::CPUDeviceContext, float>::Compute(paddle::framework::ExecutionContext const&) const
7 paddle::operators::math::SetConstant<paddle::platform::CPUDeviceContext, unsigned char>::operator()(paddle::platform::CPUDeviceContext const&, paddle::framework::Tensor*, unsigned char)
8 unsigned char* paddle::framework::Tensor::data()
9 paddle::framework::Tensor::check_memory_size() const
10 paddle::platform::EnforceNotMet::EnforceNotMet(paddle::platform::ErrorSummary const&, char const*, int)
11 paddle::platform::GetCurrentTraceBackString(bool)

Error Message Summary:

PreconditionNotMetError: Tensor's dimension is out of bound.Tensor's dimension must be equal or less than the size of its memory.But received Tensor's dimension is d5120 memory's size is 384.
[Hint: Expected numel() * SizeOfType(type()) <= memory_size(), but received numel() * SizeOfType(type()):5120 > memory_size():384.] (at /home/wangye19/Tag_BaiduGcc82_mkl_Compile_D/Paddle/paddle/fluid/framework/tensor.cc:39)
[operator < rnn > error]

pkln4tw6

pkln4tw61#

您好,我们已经收到了您的问题,会安排技术人员尽快解答您的问题,请耐心等待。请您再次检查是否提供了清晰的问题描述、复现代码、环境&版本、报错信息等。同时,您也可以通过查看 官网API文档常见问题历史IssueAI社区 来寻求解答。祝您生活愉快~

Hi! We've received your issue and please be patient to get responded. We will arrange technicians to answer your questions as soon as possible. Please make sure that you have posted enough message to demo your request. You may also check out the APIFAQGithub Issue and AI community to get the answer.Have a nice day!

6psbrbz9

6psbrbz92#

你好,已经看到你的问题。
根据报错信息,判断是高并发导致分配的内存不够用于保存接收到的数据,具体原因及解决方案需要再分析。
请问方便提供一份最小量可复现问题的代码吗?

8zzbczxx

8zzbczxx3#

你好. config配置了这两个优化项, 目前看高qps能扛得住了, 能解释下这两行干了什么吗, 为什么不加就崩了呢 = =

// 设置 MKLDNN 的 cache 容量大小
cig.SetMkldnnCacheCapacity(1);
// 开启内存优化
cig.EnableMemoryOptim();
d4so4syb

d4so4syb4#

请问这个config 是什么?有相关API吗

相关问题