pytorch 运行时错误:CUDA错误:device-side assert triggered -当第二次调用模型时

nzkunb0c  于 2022-11-09  发布在  其他
关注(0)|答案(1)|浏览(97)

我在使用PyTorch模型时遇到以下错误:

/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   2197         # remove once script supports set_grad_enabled
   2198         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 2199     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   2200 
   2201 

RuntimeError: CUDA error: device-side assert triggered

这个错误似乎只在我第二次调用模型时发生。

epochs =  500
losses = []
model.to(device)

for e in range(epochs):
  running_loss = 0
  current_batch = 1

  for x1, x2, y in data_loader:    
    print("x1 to device")
    x3 = x1.to(device)
    print("--- Computing embedding1 ---")
    embedding1 = model(x3, pooling_method=pooling_method)
    print(embedding1.size())

    print("x2 to device")
    x4 = x2.to(device)
    print("--- Computing embedding2 ---")
    embedding2 = model(x4, pooling_method=pooling_method)
    print(embedding2.size())

输出:

x1 to device
--- Computing embedding1 ---
torch.Size([64, 768])
x2 to device
--- Computing embedding2 ---
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-29-6b36cff704b2> in <module>
     21     x4 = x2.to(device)
     22     print("--- Computing embedding2 ---")
---> 23     embedding2 = model(x4, pooling_method=pooling_method)
     24     print(embedding2.size())
     25 

8 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   2197         # remove once script supports set_grad_enabled
   2198         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 2199     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   2200 
   2201 

RuntimeError: CUDA error: device-side assert triggered

输入具有相同的形状,所以问题不在于形状。错误似乎发生在模型计算输出时,但只发生在第二次。
该器械为:

device(type='cuda', index=0)

如有必要,模型为:

class BERT(nn.Module):
    """
    Torch model based on CamemBERT, in order to make sentence embeddings
    """
    def __init__(self, tokenizer, model_name=model_name, output_size=100):
        super().__init__()

        self.bert = CamembertModel.from_pretrained(model_name)
        self.bert.resize_token_embeddings(len(tokenizer))

    def forward(self, x, pooling_method='cls'):
        hidden_states = self.bert(x).last_hidden_state
        embedding = pooling(hidden_states, pooling_method=pooling_method)

        return embedding

有人知道怎么解决这个问题吗?

f4t66c6m

f4t66c6m1#

以下两个原因会导致CUDA错误发生:
1.标签/类的数量与输出单位的数量不一致:在您的情况下,中可以是嵌入大小的输入/输出。
1.损失函数的输入可能不正确:不确定你使用的是什么损失,或者你是否在BERT中改变了它的默认值。
在此处查看解决方案--〉https://builtin.com/software-engineering-perspectives/cuda-error-device-side-assert-triggered

相关问题