华为的ascend 910机器，跑的是paddleocr的文本识别训练。
运行命令：
/home/ma-user/anaconda3/envs/MindSpore/bin/python -u train.py -c configs/rec/ch_ppocr_v2.0/rec_chinese_common_train_v2.0.yml -o use_npu=True
系统：
Linux notebook-2a7fcf5e-f8cc-473f-9414-a6ff8ebbf05a 4.19.36-vhulk1907.1.0.h619.eulerosv2r8.aarch64 #1 SMP Mon Jul 22 00:00:00 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux
Paddle：
paddlepaddle_npu-0.0.0-cp37-cp37m-linux_aarch64.whl
完整报错：
[2022/10/28 12:52:28] ppocr INFO: fc_decay : 4e-05
[2022/10/28 12:52:28] ppocr INFO: name : CTCHead
[2022/10/28 12:52:28] ppocr INFO: Neck :
[2022/10/28 12:52:28] ppocr INFO: encoder_type : rnn
[2022/10/28 12:52:28] ppocr INFO: hidden_size : 256
[2022/10/28 12:52:28] ppocr INFO: name : SequenceEncoder
[2022/10/28 12:52:28] ppocr INFO: Transform : None
[2022/10/28 12:52:28] ppocr INFO: algorithm : CRNN
[2022/10/28 12:52:28] ppocr INFO: model_type : rec
[2022/10/28 12:52:28] ppocr INFO: Eval :
[2022/10/28 12:52:28] ppocr INFO: dataset :
[2022/10/28 12:52:28] ppocr INFO: data_dir : /home/ma-user/work/text_renderer/output/
[2022/10/28 12:52:28] ppocr INFO: label_file_list : ['/home/ma-user/work/text_renderer/output/label_dev.txt']
[2022/10/28 12:52:28] ppocr INFO: name : SimpleDataSet
[2022/10/28 12:52:28] ppocr INFO: transforms :
[2022/10/28 12:52:28] ppocr INFO: DecodeImage :
[2022/10/28 12:52:28] ppocr INFO: channel_first : False
[2022/10/28 12:52:28] ppocr INFO: img_mode : BGR
[2022/10/28 12:52:28] ppocr INFO: CTCLabelEncode : None
[2022/10/28 12:52:28] ppocr INFO: RecResizeImg :
[2022/10/28 12:52:28] ppocr INFO: image_shape : [3, 32, 256]
[2022/10/28 12:52:28] ppocr INFO: KeepKeys :
[2022/10/28 12:52:28] ppocr INFO: keep_keys : ['image', 'label', 'length']
[2022/10/28 12:52:28] ppocr INFO: loader :
[2022/10/28 12:52:28] ppocr INFO: batch_size_per_card : 256
[2022/10/28 12:52:28] ppocr INFO: drop_last : False
[2022/10/28 12:52:28] ppocr INFO: num_workers : 8
[2022/10/28 12:52:28] ppocr INFO: shuffle : False
[2022/10/28 12:52:28] ppocr INFO: Global :
[2022/10/28 12:52:28] ppocr INFO: cal_metric_during_train : True
[2022/10/28 12:52:28] ppocr INFO: character_dict_path : /home/ma-user/work/text_renderer/keys.txt
[2022/10/28 12:52:28] ppocr INFO: checkpoints : None
[2022/10/28 12:52:28] ppocr INFO: distributed : False
[2022/10/28 12:52:28] ppocr INFO: epoch_num : 100
[2022/10/28 12:52:28] ppocr INFO: eval_batch_step : [0, 200]
[2022/10/28 12:52:28] ppocr INFO: infer_img : doc/imgs_words/ch/word_1.jpg
[2022/10/28 12:52:28] ppocr INFO: infer_mode : False
[2022/10/28 12:52:28] ppocr INFO: log_smooth_window : 20
[2022/10/28 12:52:28] ppocr INFO: max_text_length : 15
[2022/10/28 12:52:28] ppocr INFO: pretrained_model : ./pretrain_models/ResNet18_vd_pretrained.pdparams
[2022/10/28 12:52:28] ppocr INFO: print_batch_step : 100
[2022/10/28 12:52:28] ppocr INFO: save_epoch_step : 2000
[2022/10/28 12:52:28] ppocr INFO: save_inference_dir : None
[2022/10/28 12:52:28] ppocr INFO: save_model_dir : ./output/rec_chinese_common_v2.0
[2022/10/28 12:52:28] ppocr INFO: save_res_path : ./output/rec/predicts_chinese_common_v2.0.txt
[2022/10/28 12:52:28] ppocr INFO: use_gpu : False
[2022/10/28 12:52:28] ppocr INFO: use_npu : True
[2022/10/28 12:52:28] ppocr INFO: use_space_char : True
[2022/10/28 12:52:28] ppocr INFO: use_visualdl : False
[2022/10/28 12:52:28] ppocr INFO: Loss :
[2022/10/28 12:52:28] ppocr INFO: name : CTCLoss
[2022/10/28 12:52:28] ppocr INFO: Metric :
[2022/10/28 12:52:28] ppocr INFO: main_indicator : acc
[2022/10/28 12:52:28] ppocr INFO: name : RecMetric
[2022/10/28 12:52:28] ppocr INFO: Optimizer :
[2022/10/28 12:52:28] ppocr INFO: beta1 : 0.9
[2022/10/28 12:52:28] ppocr INFO: beta2 : 0.999
[2022/10/28 12:52:28] ppocr INFO: lr :
[2022/10/28 12:52:28] ppocr INFO: learning_rate : 0.001
[2022/10/28 12:52:28] ppocr INFO: name : Cosine
[2022/10/28 12:52:28] ppocr INFO: warmup_epoch : 5
[2022/10/28 12:52:28] ppocr INFO: name : Adam
[2022/10/28 12:52:28] ppocr INFO: regularizer :
[2022/10/28 12:52:28] ppocr INFO: factor : 4e-05
[2022/10/28 12:52:28] ppocr INFO: name : L2
[2022/10/28 12:52:28] ppocr INFO: PostProcess :
[2022/10/28 12:52:28] ppocr INFO: name : CTCLabelDecode
[2022/10/28 12:52:28] ppocr INFO: Train :
[2022/10/28 12:52:28] ppocr INFO: dataset :
[2022/10/28 12:52:28] ppocr INFO: data_dir : /home/ma-user/work/text_renderer/output/
[2022/10/28 12:52:28] ppocr INFO: label_file_list : ['/home/ma-user/work/text_renderer/output/label_train.txt']
[2022/10/28 12:52:28] ppocr INFO: name : SimpleDataSet
[2022/10/28 12:52:28] ppocr INFO: transforms :
[2022/10/28 12:52:28] ppocr INFO: DecodeImage :
[2022/10/28 12:52:28] ppocr INFO: channel_first : False
[2022/10/28 12:52:28] ppocr INFO: img_mode : BGR
[2022/10/28 12:52:28] ppocr INFO: RecAug : None
[2022/10/28 12:52:28] ppocr INFO: CTCLabelEncode : None
[2022/10/28 12:52:28] ppocr INFO: RecResizeImg :
[2022/10/28 12:52:28] ppocr INFO: image_shape : [3, 32, 256]
[2022/10/28 12:52:28] ppocr INFO: KeepKeys :
[2022/10/28 12:52:28] ppocr INFO: keep_keys : ['image', 'label', 'length']
[2022/10/28 12:52:28] ppocr INFO: loader :
[2022/10/28 12:52:28] ppocr INFO: batch_size_per_card : 256
[2022/10/28 12:52:28] ppocr INFO: drop_last : True
[2022/10/28 12:52:28] ppocr INFO: num_workers : 8
[2022/10/28 12:52:28] ppocr INFO: shuffle : True
[2022/10/28 12:52:28] ppocr INFO: profiler_options : None
[2022/10/28 12:52:28] ppocr INFO: use_npu : True
[2022/10/28 12:52:28] ppocr INFO: train with paddle 0.0.0 and device Place(npu:0)
[2022/10/28 12:52:28] ppocr INFO: Initialize indexs of datasets:['/home/ma-user/work/text_renderer/output/label_train.txt']
[2022/10/28 12:52:30] ppocr INFO: Initialize indexs of datasets:['/home/ma-user/work/text_renderer/output/label_dev.txt']
[2022/10/28 12:53:14] ppocr INFO: train dataloader has 2539 iters
[2022/10/28 12:53:14] ppocr INFO: valid dataloader has 196 iters
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params stem.0.conv.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params stem.0.bn.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params stem.0.bn.bias not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params stem.0.bn._mean not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params stem.0.bn._variance not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params stem.1.conv.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params stem.1.bn.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params stem.1.bn.bias not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params stem.1.bn._mean not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params stem.1.bn._variance not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params stem.2.conv.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params stem.2.bn.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params stem.2.bn.bias not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params stem.2.bn._mean not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params stem.2.bn._variance not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.0.conv0.conv.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.0.conv0.bn.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.0.conv0.bn.bias not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.0.conv0.bn._mean not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.0.conv0.bn._variance not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.0.conv1.conv.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.0.conv1.bn.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.0.conv1.bn.bias not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.0.conv1.bn._mean not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.0.conv1.bn._variance not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.0.short.conv.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.0.short.bn.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.0.short.bn.bias not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.0.short.bn._mean not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.0.short.bn._variance not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.1.conv0.conv.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.1.conv0.bn.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.1.conv0.bn.bias not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.1.conv0.bn._mean not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.1.conv0.bn._variance not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.1.conv1.conv.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.1.conv1.bn.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.1.conv1.bn.bias not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.1.conv1.bn._mean not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.1.conv1.bn._variance not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.2.conv0.conv.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.2.conv0.bn.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.2.conv0.bn.bias not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.2.conv0.bn._mean not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.2.conv0.bn._variance not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.2.conv1.conv.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.2.conv1.bn.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.2.conv1.bn.bias not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.2.conv1.bn._mean not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.2.conv1.bn._variance not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.2.short.conv.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.2.short.bn.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.2.short.bn.bias not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.2.short.bn._mean not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.2.short.bn._variance not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.3.conv0.conv.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.3.conv0.bn.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.3.conv0.bn.bias not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.3.conv0.bn._mean not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.3.conv0.bn._variance not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.3.conv1.conv.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.3.conv1.bn.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.3.conv1.bn.bias not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.3.conv1.bn._mean not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.3.conv1.bn._variance not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.4.conv0.conv.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.4.conv0.bn.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.4.conv0.bn.bias not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.4.conv0.bn._mean not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.4.conv0.bn._variance not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.4.conv1.conv.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.4.conv1.bn.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.4.conv1.bn.bias not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.4.conv1.bn._mean not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.4.conv1.bn._variance not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.4.short.conv.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.4.short.bn.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.4.short.bn.bias not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.4.short.bn._mean not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.4.short.bn._variance not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.5.conv0.conv.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.5.conv0.bn.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.5.conv0.bn.bias not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.5.conv0.bn._mean not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.5.conv0.bn._variance not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.5.conv1.conv.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.5.conv1.bn.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.5.conv1.bn.bias not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.5.conv1.bn._mean not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.5.conv1.bn._variance not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.6.conv0.conv.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.6.conv0.bn.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.6.conv0.bn.bias not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.6.conv0.bn._mean not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.6.conv0.bn._variance not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.6.conv1.conv.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.6.conv1.bn.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.6.conv1.bn.bias not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.6.conv1.bn._mean not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.6.conv1.bn._variance not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.6.short.conv.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.6.short.bn.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.6.short.bn.bias not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.6.short.bn._mean not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.6.short.bn._variance not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.7.conv0.conv.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.7.conv0.bn.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.7.conv0.bn.bias not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.7.conv0.bn._mean not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.7.conv0.bn._variance not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.7.conv1.conv.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.7.conv1.bn.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.7.conv1.bn.bias not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.7.conv1.bn._mean not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params blocks.7.conv1.bn._variance not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params fc.weight not in model
[2022/10/28 12:53:15] ppocr WARNING: The pretrained params fc.bias not in model
[2022/10/28 12:53:15] ppocr INFO: load pretrain successful from ./pretrain_models/ResNet18_vd_pretrained
[2022/10/28 12:53:15] ppocr INFO: During the training process, after the 0th iteration, an evaluation is run every 200 iterations
Traceback (most recent call last):
File "train.py", line 208, in
main(config, device, logger, vdl_writer)
File "train.py", line 183, in main
amp_level, amp_custom_black_list)
File "/home/ma-user/work/PaddleOCR/tools/program.py", line 297, in train
avg_loss.backward()
File "</home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/decorator.py:decorator-gen-158>", line 2, in backward
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/paddle/fluid/wrapped_decorator.py", line 25, in impl
return wrapped_func(*args, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/paddle/fluid/framework.py", line 529, in impl
return func(*args, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/paddle/fluid/dygraph/varbase_patch_methods.py", line 321, in backward
framework._dygraph_tracer(),
ValueError: (InvalidArgument) AvgPoolGrad dose not support Asymmetric strides. but strides = (2, 1) (at /home/ma-user/Paddle/paddle/fluid/operators/pool_op_npu.cc:313)