嘿，这是我目前捕获的内容。我将模型存储在~/meta-llama-3.1-8b-instruct。

vector jiff ~/repo $ git clone git@github.com:meta-llama/llama-recipes.git
vector jiff ~/repo $ git clone git@github.com:huggingface/transformers.git
vector jiff ~/repo $ cd transformers
vector jiff ~/repo/transformers $ python3 -m venv .
vector jiff ~/repo/transformers $ source bin/activate
(transformers) vector jiff ~/repo/transformers $ pip install -r ../llama-recipes/requirements.txt
(transformers) vector jiff ~/repo/transformers $ pip install protobuf blobfile
(transformers) vector jiff ~/repo/transformers $ python3 src/transformers/models/llama/convert_llama_weights_to_hf.py --input_dir ~/meta-llama-3.1-8b-instruct --model_size 8B --llama_version 3.1 --output_dir ~/meta-llama-3.1-8b-instruct

Source
接下来，您还需要从Hugging Face仓库中提取一些文件到模型目录。

vector jiff ~/meta-llama-3.1-8b-instruct $ git init .
vector jiff ~/meta-llama-3.1-8b-instruct $ git remote add origin https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct
vector jiff ~/meta-llama-3.1-8b-instruct $ GIT_LFS_SKIP_SMUDGE=1 git fetch
vector jiff ~/meta-llama-3.1-8b-instruct $ git checkout origin/main -- config.json generation_config.json special_tokens_map.json tokenizer.json tokenizer_config.json

最后，按照指示运行量化步骤。

vector jiff ~/repo $ git clone https://github.com/ggerganov/llama.cpp
vector jiff ~/repo $ cd llama.cpp
vector jiff ~/repo/llama.cpp $ git log -1 --pretty=format:"%H - %s" origin/HEAD
afbb4c1322a747d2a7b4bf67c868148f8afcc6c8 - ggml-cuda: Adding support for unified memory (#8035)
vector jiff ~/repo/llama.cpp $ python3 -m venv .
vector jiff ~/repo/llama.cpp $ source bin/activate
(llama.cpp) vector jiff ~/repo/llama.cpp $ pip install -r requirements.txt
(llama.cpp) vector jiff ~/repo/llama.cpp $ pip install transformers~=4.43.3
(llama.cpp) vector jiff ~/repo/llama.cpp $ ln -s ~/meta-llama-3.1-8b-instruct models/meta-llama-3.1-8b-instruct 
(llama.cpp) vector jiff ~/repo/llama.cpp $ python3 convert_hf_to_gguf.py --outtype bf16 models/meta-llama-3.1-8b-instruct --outfile models/meta-llama-3.1-8b-instruct/meta-llama-3.1-8b-instruct-bf16.gguf

不幸的是，我在这一点上遇到了以下错误。我还没有解决这个问题。

Traceback (most recent call last):
  File "/home/jiff/predictors/repos/llama.cpp/convert_hf_to_gguf.py", line 3717, in <module>
    main()
  File "/home/jiff/predictors/repos/llama.cpp/convert_hf_to_gguf.py", line 3711, in main
    model_instance.write()
  File "/home/jiff/predictors/repos/llama.cpp/convert_hf_to_gguf.py", line 401, in write
    self.prepare_metadata(vocab_only=False)
  File "/home/jiff/predictors/repos/llama.cpp/convert_hf_to_gguf.py", line 394, in prepare_metadata
    self.set_vocab()
  File "/home/jiff/predictors/repos/llama.cpp/convert_hf_to_gguf.py", line 1470, in set_vocab
    self._set_vocab_sentencepiece()
  File "/home/jiff/predictors/repos/llama.cpp/convert_hf_to_gguf.py", line 693, in _set_vocab_sentencepiece
    tokens, scores, toktypes = self._create_vocab_sentencepiece()
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jiff/predictors/repos/llama.cpp/convert_hf_to_gguf.py", line 713, in _create_vocab_sentencepiece
    tokenizer.LoadFromFile(str(tokenizer_path))
  File "/home/jiff/predictors/repos/llama.cpp/lib/python3.11/site-packages/sentencepiece/__init__.py", line 316, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Internal: could not parse ModelProto from models/meta-llama-3.1-8b-instruct/tokenizer.model

你缺少一些Python包(protobuf?)。使用pip install将其拉入。

你描述的情况是我从https://voorloopnul.com/blog/quantize-and-run-the-original-llama3-8b-with-llama-cpp/中收集到的，我设法得到了一个可用的gguf文件。我也可以毫无问题地量化到Q8_0,并且我有一套似乎运行正常的模型。我只是不太确定绳索缩放是否能正确处理高达128k的上下文缓冲区。

鉴于在类似问题https://www.reddit.com/r/LocalLLaMA/comments/1eeyw0v/i_keep_getting_this_error_in_llama_31_8b_llamacpp/上的一些评论，例如，我在转换后的Tensor数量总是为291个，而从未有人抱怨llama.cpp期望有292个(正确转换的模型将有这个额外的一个)。这让我感到困惑。

• 我们需要通过上述过程(上面或voorloop链接)使用特殊标志/参数吗？
• 量化需要特殊标志吗？
• llama-cli或llama-server调用需要特殊标志/参数吗？

如前所述，我希望在llama.cpp中正确使用官方Meta Llama 3.1(Instruct)权重的地方看到一份官方描述。也许还可以谈谈如何测试并确保大型上下文正常工作。

@xocite,我遇到了类似的问题。发现可能是由于缺少配置文件导致的。具体来说，是tokenizer_config.json文件？我没有完全尝试解决这个问题，但我确实创建了一个gist,其中我使用了save_pretrained方法下载了tokenizer和tokenizer_config JSON文件：https://gist.github.com/brandenvs/2dad0e864fc2a0a5ee176213ae43b902

我按照你的指示运行得很好。