错误Azure Synapse笔记本:导入错误:此模块正常运行需要标记化器>= 0.10.1,〈0.11,但找到的标记化器== 0.9.2

0sgqnhkj  于 2023-01-14  发布在  其他
关注(0)|答案(2)|浏览(208)

我正在尝试在Azure synapse笔记本中执行以下命令。

from transformers import AutoTokenizer, AutoModelForMaskedLM 
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModelForMaskedLM.from_pretrained("bert-base-uncased")

然而,我得到下面的错误.

ImportError: tokenizers>=0.10.1,<0.11 is required for a normal functioning of this module, but found tokenizers==0.9.2.
Try: pip install transformers -U or pip install -e '.[dev]' if you're working with git master
Traceback (most recent call last):

  File "/home/trusted-service-user/cluster-env/env/lib/python3.6/site-packages/transformers/__init__.py", line 43, in <module>
    from . import dependency_versions_check

  File "/home/trusted-service-user/cluster-env/env/lib/python3.6/site-packages/transformers/dependency_versions_check.py", line 41, in <module>
    require_version_core(deps[pkg])

  File "/home/trusted-service-user/cluster-env/env/lib/python3.6/site-packages/transformers/utils/versions.py", line 120, in require_version_core
    return require_version(requirement, hint)

  File "/home/trusted-service-user/cluster-env/env/lib/python3.6/site-packages/transformers/utils/versions.py", line 114, in require_version
    _compare_versions(op, got_ver, want_ver, requirement, pkg, hint)

  File "/home/trusted-service-user/cluster-env/env/lib/python3.6/site-packages/transformers/utils/versions.py", line 50, in _compare_versions
    f"{requirement} is required for a normal functioning of this module, but found {pkg}=={got_ver}.{hint}"

ImportError: tokenizers>=0.10.1,<0.11 is required for a normal functioning of this module, but found tokenizers==0.9.2.
Try: pip install transformers -U or pip install -e '.[dev]' if you're working with git master

即使在我升级了转换器和令牌化器版本并通过下面的代码验证之后,上面的错误仍然存在:

import pkg_resources
for d in pkg_resources.working_set:
    print(d)

以上代码的输出示例:

transformers 4.10.3
tokenizers 0.10.3

有什么想法或建议吗?先谢了。

8fsztsew

8fsztsew1#

你可以在你的控制台中尝试这个命令:

pip install -I transformers --no-cache-dir --force-reinstall
gupuwyp2

gupuwyp22#

使用

pip list | grep packaging

检查您的软件包版本是否高于21.3
尝试重新安装你的软件包,就像这样

pip install packaging==21.3

此问题与新打包版本检查中的格式要求有关
我想我应该添加一个评论,而不是添加一个答案在这里,因为这个答案可能不会解决你的问题,但我没有足够的声誉,所以对不起。

相关问题