DeepSpeed-MII 如何在管道中设置trust_remote_code=True?

c7rzv4ha  于 6个月前  发布在  其他
关注(0)|答案(2)|浏览(96)

$Hi, how to set trust_remote_code=True in mii pipeline?

Code

from mii import pipeline
pipe = pipeline("/root/Qwen-7B-Chat")
pipe.tokenizer.tokenizer.eos_token_id = 151643
output = pipe(["DeepSpeed is"], max_new_tokens=128, do_sample=False)
print(output)

Output

The repository for /root/Qwen-7B-Chat contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co//root/Qwen-7B-Chat.
You can avoid this prompt in future by passing the argument `trust_remote_code=True`.

Do you wish to run the custom code? [y/N]
vcirk6k6

vcirk6k61#

我认为目前没有暴露的接口来设置'trust_remote_code'参数。您可以通过直接修改deepspeed源代码中的load方法来设置它。
https://github.com/microsoft/DeepSpeed/blob/2a6c58df34ea04e0aa451a4623b7c7d49b087a50/deepspeed/inference/v2/checkpoint/huggingface_engine.py#L24

1u4esq0p

1u4esq0p2#

Hi @gujingit,不幸的是@ZonePG是正确的。我认为这个添加已经迟到了。让我整理一个PR,将hf_kwargs选项添加到我们的API中。这应该允许传递任意参数来从HuggingFace加载检查点权重。

相关问题