这个问题与How can I check a confusion_matrix after fine-tuning with custom datasets?相同,在Data Science Stack Exchange上。
背景
我想检查一个confusion_matrix,包括precision,recall和f1-score,如下所示,经过自定义数据集的微调。
微调过程和任务是序列分类与IMDb评论上的自定义数据集微调教程拥抱脸。
使用Trainer完成微调后,在这种情况下如何检查confusion_matrix?
confusion_matrix的图像,包括precision、recall和f1-score original site:仅作为输出图像示例
predictions = np.argmax(trainer.test(test_x), axis=1)
# Confusion matrix and classification report.
print(classification_report(test_y, predictions))
precision recall f1-score support
0 0.75 0.79 0.77 1000
1 0.81 0.87 0.84 1000
2 0.63 0.61 0.62 1000
3 0.55 0.47 0.50 1000
4 0.66 0.66 0.66 1000
5 0.62 0.64 0.63 1000
6 0.74 0.83 0.78 1000
7 0.80 0.74 0.77 1000
8 0.85 0.81 0.83 1000
9 0.79 0.80 0.80 1000
avg / total 0.72 0.72 0.72 10000
代码
from transformers import DistilBertForSequenceClassification, Trainer, TrainingArguments
training_args = TrainingArguments(
output_dir='./results', # output directory
num_train_epochs=3, # total number of training epochs
per_device_train_batch_size=16, # batch size per device during training
per_device_eval_batch_size=64, # batch size for evaluation
warmup_steps=500, # number of warmup steps for learning rate scheduler
weight_decay=0.01, # strength of weight decay
logging_dir='./logs', # directory for storing logs
logging_steps=10,
)
model = DistilBertForSequenceClassification.from_pretrained("distilbert-base-uncased")
trainer = Trainer(
model=model, # the instantiated 🤗 Transformers model to be trained
args=training_args, # training arguments, defined above
train_dataset=train_dataset, # training dataset
eval_dataset=val_dataset # evaluation dataset
)
trainer.train()
到目前为止我所做的
数据集准备序列分类与IMDb评论,我微调与教练。
from pathlib import Path
def read_imdb_split(split_dir):
split_dir = Path(split_dir)
texts = []
labels = []
for label_dir in ["pos", "neg"]:
for text_file in (split_dir/label_dir).iterdir():
texts.append(text_file.read_text())
labels.append(0 if label_dir is "neg" else 1)
return texts, labels
train_texts, train_labels = read_imdb_split('aclImdb/train')
test_texts, test_labels = read_imdb_split('aclImdb/test')
from sklearn.model_selection import train_test_split
train_texts, val_texts, train_labels, val_labels = train_test_split(train_texts, train_labels, test_size=.2)
from transformers import DistilBertTokenizerFast
tokenizer = DistilBertTokenizerFast.from_pretrained('distilbert-base-uncased')
train_encodings = tokenizer(train_texts, truncation=True, padding=True)
val_encodings = tokenizer(val_texts, truncation=True, padding=True)
test_encodings = tokenizer(test_texts, truncation=True, padding=True)
import torch
class IMDbDataset(torch.utils.data.Dataset):
def __init__(self, encodings, labels):
self.encodings = encodings
self.labels = labels
def __getitem__(self, idx):
item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
item['labels'] = torch.tensor(self.labels[idx])
return item
def __len__(self):
return len(self.labels)
train_dataset = IMDbDataset(train_encodings, train_labels)
val_dataset = IMDbDataset(val_encodings, val_labels)
test_dataset = IMDbDataset(test_encodings, test_labels)
1条答案
按热度按时间u7up0aaq1#
在这种情况下,您可以做的是迭代验证集(或者测试集),并手动创建
y_true
和y_pred
的列表。最后,
观察结果:
1.模型的输出是
logits
,而不是归一化的概率。1.因此,我们在维度1上应用
softmax
以变换为实际概率(例如,0.2% class 0
,0.8% class 1
)。1.我们应用
.argmax()
操作来获取类的索引。