AdamW的实现已弃用,并将在未来版本中删除,请使用PyTorch实现torch.optim.AdamW

nmpmafwu  于 2023-03-02  发布在  其他
关注(0)|答案(1)|浏览(540)

如何修复这个过时的AdamW模型?
我尝试使用BERT模型对酒店评论进行情感分析,当我运行这段代码时,它提示了以下警告。我还在研究变形金刚,我不希望代码很快被弃用。我在网上搜索了一下,还没有找到解决方案。
我找到了这条信息,但我不知道如何将其应用到我的代码中。
要切换优化器,请将optim=“adamw_torch”放入您的TrainingArguments(默认值为“adamw_hf”)
有谁能帮忙吗?

from transformers import BertTokenizer, BertForSequenceClassification
import torch_optimizer as optim
from torch.utils.data import DataLoader
from transformers import AdamW
import pandas as pd
import torch
import random
import numpy as np
import torch.nn as nn
from torch.nn import CrossEntropyLoss
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, confusion_matrix, roc_auc_score, classification_report
from sklearn.model_selection import train_test_split
from sklearn import preprocessing
from tqdm.notebook import tqdm
import json
from collections import OrderedDict
import logging
from torch.utils.tensorboard import SummaryWriter

跳过一些代码...

param_optimizer = list(model.named_parameters())
no_decay = ['bias', 'gamma', 'beta']
optimizer_grouped_parameters = [{
    'params':
    [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
    'weight_decay_rate':
    0.01
}, {
    'params':
    [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],
    'weight_decay_rate':
    0.0
}]

#
optimizer = AdamW(optimizer_grouped_parameters, lr=1e-5)           ##deprecated 
#optimizer = optim.AdamW(optimizer_grouped_parameters, lr=1e-5)      ##torch.optim.AdamW  (not working)

step = 0
best_acc = 0
epoch = 10
writer = SummaryWriter(log_dir='model_best')
for epoch in tqdm(range(epoch)):
    for idx, batch in tqdm(enumerate(train_loader),
                           total=len(train_texts) // batch_size,
                           leave=False):
        optimizer.zero_grad()
        input_ids = batch['input_ids'].to(device)
        attention_mask = batch['attention_mask'].to(device)
        labels = batch['labels'].to(device)
        outputs = model(input_ids=input_ids, attention_mask=attention_mask, labels=labels)
        loss = outputs[0]  # Calculate Loss
        logging.info(
            f'Epoch-{epoch}, Step-{step}, Loss: {loss.cpu().detach().numpy()}')
        step += 1
        loss.backward()
        optimizer.step()
        writer.add_scalar('train_loss', loss.item(), step)
    logging.info(f'Epoch {epoch}, present best acc: {best_acc}, start evaluating.')
    accuracy, precision, recall, f1 = eval_model(model, eval_loader)  # Evaluate Model
    writer.add_scalar('dev_accuracy', accuracy, step)
    writer.add_scalar('dev_precision', precision, step)
    writer.add_scalar('dev_recall', recall, step)
    writer.add_scalar('dev_f1', f1, step)
    if accuracy > best_acc:
        model.save_pretrained('model_best')  # Save Model
        tokenizer.save_pretrained('model_best')
        best_acc = accuracy
rkttyhzu

rkttyhzu1#

如果注解掉这两行:

import torch_optimizer as optim
from transformers import AdamW

然后使用:

optimizer = torch.optim.AdamW(optimizer_grouped_parameters, lr=1e-5)

是否有效?如果无效,错误是什么?
要切换优化器,请将optim ="adamw_torch"放入您的TrainingArguments(默认值为"adamw_hf")
这是指Huggingface Trainer,它配置了一个TrainingArguments示例,但由于您使用的是自己的训练循环,因此不适用于您。

相关问题