tensorflow 简单转换器错误:版本恩怨:标记化器==0.9.4?我该如何解决此问题?

djmepvbi  于 2023-08-06  发布在  其他
关注(0)|答案(3)|浏览(118)

我试着从google colab的网站上执行simpletransformers的例子。
范例:

from simpletransformers.classification import ClassificationModel, ClassificationArgs
import pandas as pd
import logging

logging.basicConfig(level=logging.INFO)
transformers_logger = logging.getLogger("transformers")
transformers_logger.setLevel(logging.WARNING)

# Preparing train data
train_data = [
    ["Aragorn was the heir of Isildur", 1],
    ["Frodo was the heir of Isildur", 0],
]
train_df = pd.DataFrame(train_data)
train_df.columns = ["text", "labels"]

# Preparing eval data
eval_data = [
    ["Theoden was the king of Rohan", 1],
    ["Merry was the king of Rohan", 0],
]
eval_df = pd.DataFrame(eval_data)
eval_df.columns = ["text", "labels"]

# Optional model configuration
model_args = ClassificationArgs(num_train_epochs=1)

# Create a ClassificationModel
model = ClassificationModel(
    "roberta", "roberta-base", args=model_args
)

# Train the model
model.train_model(train_df)

# Evaluate the model
result, model_outputs, wrong_predictions = model.eval_model(eval_df)

# Make predictions with the model
predictions, raw_outputs = model.predict(["Sam was a Wizard"])

字符串
但它给了我以下错误:
版本恩怨:此模块的正常运行需要标记器==0.9.4,但发现标记器==0.10.0。如果你使用的是git master,可以尝试:pip install transformers -U或pip install -e '.[dev]'
我试过!pip install transformers -U,甚至!pip install tokenizers==0.9.4,但一直给出相同的错误。我以前执行过这段代码,它工作得很有趣,但现在它给出了上述错误。

2wnc66cl

2wnc66cl1#

我把这个放在这里,以防有人遇到同样的问题。我得到了造物主的帮助。

Workaround:
Install tokenizers==0.9.4 before install simpletransformers

In Colab for example;

!pip install tokenizers==0.9.4
!pip install simpletransformers

字符串
https://github.com/ThilinaRajapakse/simpletransformers/issues/950

jhdbpxl9

jhdbpxl92#

如上所述,我认为您基本上需要的是!pip install tokenizers==0.9.4,它将卸载tokenizers-0.10.0并安装tokenizers-0.9.4

hwazgwia

hwazgwia3#

我也面临着同样的问题。但在我的情况下,我对simpletransformers没有问题,而版本冲突是与transformers和tokenizer。
首先,卸载transformers和tokenizer,在colab中,你应该使用!之前pip如!点

pip uninstall transformers
pip uninstall tokenizers

字符串
然后,

pip install tokenizers==0.9.4 transformers==4.1.1


transformers 4.1.1与tokenizer 0.9.4兼容

相关问题