Paddle 【论文复现】多卡训练时运行时间变长的问题

fv2wmkja 于 2022-04-21 发布在 Java

关注(0)|答案(29)|浏览(606)

版本、环境信息：

1）PaddlePaddle版本：请提供您的PaddlePaddle版本号，1.8.2
2）GPU：aistudio平台的脚步任务环境（选择的4卡环境进行运行测试）
项目连接：https://aistudio.baidu.com/aistudio/clusterprojectdetail/773316（已公开）
现象：每次运行到第二个epoch时发现每次循环的运行时间都变长了一倍，运行时间包括reader读数据的时间，进一步测试发现gpu的运行时间没有变化，主要是reader读数据的时间变长了一倍，不知道是什么原因。具体现象可以看输出的日志文件：
Loss at epoch 0 step 5: [4.3775268], acc: [0.125],period: 16.51509690284729
Loss at epoch 0 step 10: [3.8490183], acc: [0.40625],period: 16.0886127948761
Loss at epoch 0 step 15: [2.9390435], acc: [0.5625],period: 16.38063144683838
Loss at epoch 0 step 20: [2.3567476], acc: [0.65625],period: 16.37914490699768
Loss at epoch 0 step 25: [1.5900159], acc: [0.734375],period: 16.665042877197266
Loss at epoch 0 step 30: [1.2181798], acc: [0.734375],period: 15.94478726387024
Loss at epoch 0 step 35: [0.6449917], acc: [0.90625],period: 16.192134141921997
valid Loss at step 0: [0.25852293], acc: [1.]
valid Loss at step 20: [0.46329147], acc: [0.875]
valid Loss at step 40: [1.2772802], acc: [0.71875]
valid Loss at step 60: [1.3179752], acc: [0.6875]
valid Loss at step 80: [1.1119821], acc: [0.75]
start data reader (trainers_num: 4, trainer_id: 1)
valid Loss at step 100: [2.2030392], acc: [0.1875]
start data reader (trainers_num: 4, trainer_id: 3)
start data reader (trainers_num: 4, trainer_id: 2)
验证集准确率为:0.8328919410705566
验证集loss为:0.7212506532669067
Epoch1 lr=0.009999999776482582
start data reader (trainers_num: 4, trainer_id: 0)
Loss at epoch 1 step 5: [0.55662304], acc: [0.890625],period: 36.16345572471619
Loss at epoch 1 step 10: [0.42250103], acc: [0.875],period: 37.54211711883545
Loss at epoch 1 step 15: [0.27849782], acc: [0.96875],period: 37.35439491271973
Loss at epoch 1 step 20: [0.33116293], acc: [0.90625],period: 35.67357039451599
Loss at epoch 1 step 25: [0.37910157], acc: [0.890625],period: 37.441500663757324
Loss at epoch 1 step 30: [0.36237404], acc: [0.953125],period: 37.42715764045715
Loss at epoch 1 step 35: [0.36434418], acc: [0.921875],period: 35.077528953552246
valid Loss at step 0: [0.08099228], acc: [1.]
valid Loss at step 20: [0.43469694], acc: [0.84375]
valid Loss at step 40: [1.1833289], acc: [0.71875]
valid Loss at step 60: [0.94777864], acc: [0.71875]
valid Loss at step 80: [0.42069796], acc: [0.9375]
valid Loss at step 100: [1.0385896], acc: [0.71875]
start data reader (trainers_num: 4, trainer_id: 2)
start data reader (trainers_num: 4, trainer_id: 3)
start data reader (trainers_num: 4, trainer_id: 1)
验证集准确率为:0.8781779408454895
验证集loss为:0.44851189851760864
Epoch2 lr=0.0010000000474974513
start data reader (trainers_num: 4, trainer_id: 0)
Loss at epoch 2 step 5: [0.38328272], acc: [0.921875],period: 37.521843671798706
Loss at epoch 2 step 10: [0.3836357], acc: [0.90625],period: 36.96732306480408
Loss at epoch 2 step 15: [0.2954501], acc: [0.9375],period: 37.78758931159973
Loss at epoch 2 step 20: [0.28290433], acc: [0.9375],period: 36.79577970504761
Loss at epoch 2 step 25: [0.23077166], acc: [0.984375],period: 35.68168759346008
Loss at epoch 2 step 30: [0.303536], acc: [0.9375],period: 37.21159076690674
Loss at epoch 2 step 35: [0.25395435], acc: [0.9375],period: 36.25998377799988
valid Loss at step 0: [0.07491596], acc: [1.]
valid Loss at step 20: [0.39159745], acc: [0.84375]
valid Loss at step 40: [1.0963948], acc: [0.71875]
valid Loss at step 60: [0.9259555], acc: [0.71875]
valid Loss at step 80: [0.5368143], acc: [0.90625]
valid Loss at step 100: [0.93665516], acc: [0.78125]
验证集准确率为:0.8916842937469482
验证集loss为:0.41894418001174927
INFO 2020-09-01 00:58:53,138 launch.py:223] Local procs complete, POD info:rank:0 id:None addr:127.0.0.1 port:None visible_gpu:[] trainers:["gpu:['0'] endpoint:127.0.0.1:52136 rank:0", "gpu:['1'] endpoint:127.0.0.1:35241 rank:1", "gpu:['2'] endpoint:127.0.0.1:47208 rank:2", "gpu:['3'] endpoint:127.0.0.1:49438 rank:3"]

period就是统计的每个iter的运行时间（含读数时间），测试代码如下:
cur_time = 0
for batch_id, data in enumerate(train_reader()):

dy_x_data = np.array([x[0] for x in data]).astype('float32')
            y_data = np.array([[x[1]] for x in data]).astype('int64')

            period = time.time()-cur_time

            cur_time = time.time()
            img = fluid.dygraph.to_variable(dy_x_data)
            label = fluid.dygraph.to_variable(y_data)
            label.stop_gradient = True

            out = train_model(img)
            acc = fluid.layers.accuracy(out,label)
            loss = fluid.layers.cross_entropy(out, label)
            avg_loss = fluid.layers.mean(loss)

            avg_loss = train_model.scale_loss(avg_loss)
            avg_loss.backward()
            train_model.apply_collective_grads()

            opt.minimize(avg_loss)
            train_model.clear_gradients()

            if(dev_id==0):
                if batch_id % 5 == 0 and batch_id>0:
                    print("Loss at epoch {} step {}: {}, acc: {},period: {}".format(i, batch_id, avg_loss.numpy()*4, acc.numpy(),period))

Paddle

来源：https://github.com/PaddlePaddle/Paddle/issues/26852