在预处理阶段，我想将非常长的序列截断为较短的序列。我发现Tokenizer首先对整个序列进行编码，然后再截断它。有没有一种方法可以在截断时实时进行，以节省对序列未使用部分的标记化时间？

我使用的是transformers 4.42.4版本，tokenizer类是LlamaTokenizerFast。以下是代码和输出：

from transformers import LlamaTokenizerFast

tokenizer = LlamaTokenizerFast()
sequence = "

from transformers import AutoTokenizer
from time import time

tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/deepseek-coder-1.3b-base")

tokenizer class is LlamaTokenizerFast

def encode_long(meta_len, batch_size=200):
text = 500*"jibber-jabber"
text_batch = batch_size*[meta_len*text]

start_time = time()
batch = tokenizer.batch_encode_plus(text_batch, truncation=True, max_length=1000)
print(f"Seq encoded len = {len(batch['input_ids'][0])}; Time used = {time() - start_time:.2f}, Symbols = {meta_len*len(text)}")

encode_long(meta_len=1)
encode_long(meta_len=10)
encode_long(meta_len=100)

" * 1000
encoded_sequence = tokenizer.encode(sequence)
print("Seq encoded len =", len(encoded_sequence), "; Time used =", tokenizer.batch_encode_plus(sequence, add_special_tokens=True)[0]['input_ids'].shape[0], "; Symbols =", len(tokenizer))

代码在我机器上的运行结果：

Seq encoded len = 1000; Time used = 0.07, Symbols = 6500
Seq encoded len = 1000; Time used = 0.59, Symbols = 65000
Seq encoded len = 1000; Time used = 8.98, Symbols = 650000