你好，
你能分享一下你使用HuggingFace transformers运行模型的代码吗？
另外，在转换并加载模型到CTranslate2时，你设置了哪些参数？

Hi,
Can you share the code you are using to run the model with HuggingFace transformers?
Also what parameters do you set when converting and then loading the model to CTranslate2?
Sure.
Here's the code:
HF Transformers

import transformers
import torch
import time

tokenizer = transformers.AutoTokenizer.from_pretrained(model_id)
chatbot = transformers.pipeline(
    "text-generation",
    model=model_id,
    torch_dtype=torch.float16,
    device_map="auto",
)

chatbot.pad_token_id=tokenizer.eos_token_id,
# Warm up the model
chatbot(
    "Who is the president of the US?",
    do_sample=False,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
    max_length=10,
)
input = llama2_chat_prompt_template.format(transcript=transcript)
start = time.time()
sequences = chatbot(
    [input, input, input],
    do_sample=False,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
    max_length=1200,
    batch_size=5
)
end = time.time()
t = end-start

FOR CTranslate2

import ctranslate2
import transformers
import time

# Load and warmup the model
start = time.time()
generator = ctranslate2.Generator("/content/Llama-2-13B-Chat-ct2", device="cuda")
tokenizer = transformers.AutoTokenizer.from_pretrained("TheBloke/Llama-2-13B-Chat-fp16")
tokens = tokenizer.convert_ids_to_tokens(tokenizer.encode("WHo is the president of the US?"))
results = generator.generate_batch([tokens], max_length=5, include_prompt_in_result=False)
output = tokenizer.decode(results[0].sequences_ids[0])
end = time.time()
t = end - start
print("Time:\t", t)
print(output)

# Run speed test
input = llama2_chat_prompt_template.format(transcript=transcript)

start = time.time()

tokens = tokenizer.convert_ids_to_tokens(tokenizer.encode(input))
results = generator.generate_batch([tokens], max_length=512, include_prompt_in_result=False)
output = tokenizer.decode(results[0].sequences_ids[0])

end = time.time()

t = end-start

Conversion script:

# Unquantized
import time
import os
start = time.time()
os.system("ct2-transformers-converter --model TheBloke/Llama-2-13B-Chat-fp16 --quantization float16 --output_dir Llama-2-13B-Chat-ct2")
end = time.time()
t = end - start
print("Time:\t", t)