AdamW的实现已弃用,并将在未来版本中删除,请使用PyTorch实现torch.optim.AdamW

nmpmafwu  于 2023-03-02  发布在  其他
关注(0)|答案(1)|浏览(618)

如何修复这个过时的AdamW模型?
我尝试使用BERT模型对酒店评论进行情感分析,当我运行这段代码时,它提示了以下警告。我还在研究变形金刚,我不希望代码很快被弃用。我在网上搜索了一下,还没有找到解决方案。
我找到了这条信息,但我不知道如何将其应用到我的代码中。
要切换优化器,请将optim=“adamw_torch”放入您的TrainingArguments(默认值为“adamw_hf”)
有谁能帮忙吗?

  1. from transformers import BertTokenizer, BertForSequenceClassification
  2. import torch_optimizer as optim
  3. from torch.utils.data import DataLoader
  4. from transformers import AdamW
  5. import pandas as pd
  6. import torch
  7. import random
  8. import numpy as np
  9. import torch.nn as nn
  10. from torch.nn import CrossEntropyLoss
  11. from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, confusion_matrix, roc_auc_score, classification_report
  12. from sklearn.model_selection import train_test_split
  13. from sklearn import preprocessing
  14. from tqdm.notebook import tqdm
  15. import json
  16. from collections import OrderedDict
  17. import logging
  18. from torch.utils.tensorboard import SummaryWriter

跳过一些代码...

  1. param_optimizer = list(model.named_parameters())
  2. no_decay = ['bias', 'gamma', 'beta']
  3. optimizer_grouped_parameters = [{
  4. 'params':
  5. [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
  6. 'weight_decay_rate':
  7. 0.01
  8. }, {
  9. 'params':
  10. [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],
  11. 'weight_decay_rate':
  12. 0.0
  13. }]
  14. #
  15. optimizer = AdamW(optimizer_grouped_parameters, lr=1e-5) ##deprecated
  16. #optimizer = optim.AdamW(optimizer_grouped_parameters, lr=1e-5) ##torch.optim.AdamW (not working)
  17. step = 0
  18. best_acc = 0
  19. epoch = 10
  20. writer = SummaryWriter(log_dir='model_best')
  21. for epoch in tqdm(range(epoch)):
  22. for idx, batch in tqdm(enumerate(train_loader),
  23. total=len(train_texts) // batch_size,
  24. leave=False):
  25. optimizer.zero_grad()
  26. input_ids = batch['input_ids'].to(device)
  27. attention_mask = batch['attention_mask'].to(device)
  28. labels = batch['labels'].to(device)
  29. outputs = model(input_ids=input_ids, attention_mask=attention_mask, labels=labels)
  30. loss = outputs[0] # Calculate Loss
  31. logging.info(
  32. f'Epoch-{epoch}, Step-{step}, Loss: {loss.cpu().detach().numpy()}')
  33. step += 1
  34. loss.backward()
  35. optimizer.step()
  36. writer.add_scalar('train_loss', loss.item(), step)
  37. logging.info(f'Epoch {epoch}, present best acc: {best_acc}, start evaluating.')
  38. accuracy, precision, recall, f1 = eval_model(model, eval_loader) # Evaluate Model
  39. writer.add_scalar('dev_accuracy', accuracy, step)
  40. writer.add_scalar('dev_precision', precision, step)
  41. writer.add_scalar('dev_recall', recall, step)
  42. writer.add_scalar('dev_f1', f1, step)
  43. if accuracy > best_acc:
  44. model.save_pretrained('model_best') # Save Model
  45. tokenizer.save_pretrained('model_best')
  46. best_acc = accuracy
rkttyhzu

rkttyhzu1#

如果注解掉这两行:

  1. import torch_optimizer as optim
  2. from transformers import AdamW

然后使用:

  1. optimizer = torch.optim.AdamW(optimizer_grouped_parameters, lr=1e-5)

是否有效?如果无效,错误是什么?
要切换优化器,请将optim ="adamw_torch"放入您的TrainingArguments(默认值为"adamw_hf")
这是指Huggingface Trainer,它配置了一个TrainingArguments示例,但由于您使用的是自己的训练循环,因此不适用于您。

相关问题