Paddle 预测部署的时候, qps高的时候报错

kh212irz 于 5个月前发布在其他

关注(0)|答案(5)|浏览(54)

版本、环境信息：
1）PaddlePaddle版本：2.2.2
2）CPU：Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
4）系统环境：CentOS release 4.3 (Final)

预测的时候, 低qps的时候请求没有问题, qps高起来会报下述错误, 请问原因有可能是什么呀:

C++ Traceback (most recent call last):

0 paddle::AnalysisPredictor::ZeroCopyRun()
1 paddle::framework::NaiveExecutor::Run()
2 paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, paddle::platform::Place const&)
3 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&) const
4 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&, paddle::framework::RuntimeContext*) const
5 std::_Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CPUPlace, false, 0ul, paddle::operators::RNNCPUKernel<paddle::platform::CPUDeviceContext, float>, paddle::operators::RNNCPUKernel<paddle::platform::CPUDeviceContext, double> >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&) #1 }>::_M_invoke(std::_Any_data const&, paddle::framework::ExecutionContext const&)
6 paddle::operators::RNNCPUKernel<paddle::platform::CPUDeviceContext, float>::Compute(paddle::framework::ExecutionContext const&) const
7 paddle::operators::math::SetConstant<paddle::platform::CPUDeviceContext, unsigned char>::operator()(paddle::platform::CPUDeviceContext const&, paddle::framework::Tensor*, unsigned char)
8 unsigned char* paddle::framework::Tensor::data()
9 paddle::framework::Tensor::check_memory_size() const
10 paddle::platform::EnforceNotMet::EnforceNotMet(paddle::platform::ErrorSummary const&, char const*, int)
11 paddle::platform::GetCurrentTraceBackString(bool)

Error Message Summary:

PreconditionNotMetError: Tensor's dimension is out of bound.Tensor's dimension must be equal or less than the size of its memory.But received Tensor's dimension is d5120 memory's size is 384.
[Hint: Expected numel() * SizeOfType(type()) <= memory_size(), but received numel() * SizeOfType(type()):5120 > memory_size():384.] (at /home/wangye19/Tag_BaiduGcc82_mkl_Compile_D/Paddle/paddle/fluid/framework/tensor.cc:39)
[operator < rnn > error]

Paddle

来源：https://github.com/PaddlePaddle/Paddle/issues/40949

5条答案

按热度按时间

pkln4tw61#

您好，我们已经收到了您的问题，会安排技术人员尽快解答您的问题，请耐心等待。请您再次检查是否提供了清晰的问题描述、复现代码、环境&版本、报错信息等。同时，您也可以通过查看官网API文档、常见问题、历史Issue 、 AI社区来寻求解答。祝您生活愉快～

Hi! We've received your issue and please be patient to get responded. We will arrange technicians to answer your questions as soon as possible. Please make sure that you have posted enough message to demo your request. You may also check out the API ， FAQ ， Github Issue and AI community to get the answer.Have a nice day!

赞(0）回复(0）举报 5个月前

6psbrbz92#

你好，已经看到你的问题。
根据报错信息，判断是高并发导致分配的内存不够用于保存接收到的数据，具体原因及解决方案需要再分析。
请问方便提供一份最小量可复现问题的代码吗？

赞(0）回复(0）举报 5个月前

8zzbczxx3#

你好. config配置了这两个优化项, 目前看高qps能扛得住了, 能解释下这两行干了什么吗, 为什么不加就崩了呢 = =

// 设置 MKLDNN 的 cache 容量大小
cig.SetMkldnnCacheCapacity(1);
// 开启内存优化
cig.EnableMemoryOptim();

赞(0）回复(0）举报 5个月前

d4so4syb4#

请问这个config 是什么？有相关API吗

赞(0）回复(0）举报 5个月前

brccelvz5#

你好, 是 paddle_infer::Config
https://paddle-inference.readthedocs.io/en/latest/api_reference/cxx_api_doc/Config/CPUConfig.html

赞(0）回复(0）举报 5个月前

我来回答

Paddle 预测部署的时候, qps高的时候报错

C++ Traceback (most recent call last):

Error Message Summary:

5条答案

相关问题

热门标签

最新问答