如何修复这个过时的AdamW模型?
我尝试使用BERT模型对酒店评论进行情感分析,当我运行这段代码时,它提示了以下警告。我还在研究变形金刚,我不希望代码很快被弃用。我在网上搜索了一下,还没有找到解决方案。
我找到了这条信息,但我不知道如何将其应用到我的代码中。
要切换优化器,请将optim=“adamw_torch”放入您的TrainingArguments(默认值为“adamw_hf”)
有谁能帮忙吗?
from transformers import BertTokenizer, BertForSequenceClassification
import torch_optimizer as optim
from torch.utils.data import DataLoader
from transformers import AdamW
import pandas as pd
import torch
import random
import numpy as np
import torch.nn as nn
from torch.nn import CrossEntropyLoss
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, confusion_matrix, roc_auc_score, classification_report
from sklearn.model_selection import train_test_split
from sklearn import preprocessing
from tqdm.notebook import tqdm
import json
from collections import OrderedDict
import logging
from torch.utils.tensorboard import SummaryWriter
跳过一些代码...
param_optimizer = list(model.named_parameters())
no_decay = ['bias', 'gamma', 'beta']
optimizer_grouped_parameters = [{
'params':
[p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
'weight_decay_rate':
0.01
}, {
'params':
[p for n, p in param_optimizer if any(nd in n for nd in no_decay)],
'weight_decay_rate':
0.0
}]
#
optimizer = AdamW(optimizer_grouped_parameters, lr=1e-5) ##deprecated
#optimizer = optim.AdamW(optimizer_grouped_parameters, lr=1e-5) ##torch.optim.AdamW (not working)
step = 0
best_acc = 0
epoch = 10
writer = SummaryWriter(log_dir='model_best')
for epoch in tqdm(range(epoch)):
for idx, batch in tqdm(enumerate(train_loader),
total=len(train_texts) // batch_size,
leave=False):
optimizer.zero_grad()
input_ids = batch['input_ids'].to(device)
attention_mask = batch['attention_mask'].to(device)
labels = batch['labels'].to(device)
outputs = model(input_ids=input_ids, attention_mask=attention_mask, labels=labels)
loss = outputs[0] # Calculate Loss
logging.info(
f'Epoch-{epoch}, Step-{step}, Loss: {loss.cpu().detach().numpy()}')
step += 1
loss.backward()
optimizer.step()
writer.add_scalar('train_loss', loss.item(), step)
logging.info(f'Epoch {epoch}, present best acc: {best_acc}, start evaluating.')
accuracy, precision, recall, f1 = eval_model(model, eval_loader) # Evaluate Model
writer.add_scalar('dev_accuracy', accuracy, step)
writer.add_scalar('dev_precision', precision, step)
writer.add_scalar('dev_recall', recall, step)
writer.add_scalar('dev_f1', f1, step)
if accuracy > best_acc:
model.save_pretrained('model_best') # Save Model
tokenizer.save_pretrained('model_best')
best_acc = accuracy
1条答案
按热度按时间rkttyhzu1#
如果注解掉这两行:
然后使用:
是否有效?如果无效,错误是什么?
要切换优化器,请将optim ="adamw_torch"放入您的TrainingArguments(默认值为"adamw_hf")
这是指Huggingface
Trainer
,它配置了一个TrainingArguments
示例,但由于您使用的是自己的训练循环,因此不适用于您。