在这个示例中,我添加了"quantization": "gptq"键值对。你可以根据实际情况修改其他键值对。修改完成后,重新运行程序,问题应该可以得到解决。 [rank0]: self.model_runner.load_model() [rank0]: File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 134, in load_model [rank0]: self.model = get_model( [rank0]: File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/init.py", line 21, in get_model [rank0]: return loader.load_model(model_config=model_config, [rank0]: File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/loader.py", line 240, in load_model [rank0]: model = _initialize_model(model_config, self.load_config, [rank0]: File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/loader.py", line 91, in _initialize_model [rank0]: return model_class(config=model_config.hf_config, [rank0]: File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/falcon.py", line 389, in init [rank0]: self.transformer = FalconModel(config, cache_config, quant_config) [rank0]: File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/falcon.py", line 350, in init [rank0]: self.h = nn.ModuleList([ [rank0]: File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/falcon.py", line 351, in [rank0]: FalconDecoderLayer(config, cache_config, quant_config) [rank0]: File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/falcon.py", line 249, in init [rank0]: if (config.num_ln_in_parallel_attn is None [rank0]: File "/home/ubuntu/.local/lib/python3.10/site-packages/transformers/configuration_utils.py", line 264, in getattribute [rank0]: return super().getattribute(key) [rank0]: AttributeError: 'RWConfig' object has no attribute 'num_ln_in_parallel_attn'
################################
I I changed the modifications here: /vllm/vllm/transformers_utils/configs$ falcon.py self.bias = bias self.parallel_attn = parallel_attn self.new_decoder_architecture = new_decoder_architecture self.num_ln_in_parallel_attn = None
if self.hidden_size == 8192:
# Hack for falcon-40b
2条答案
按热度按时间sd2nnvve1#
@sadrafh 我通过修改配置文件得到了这个模型:
然而,它似乎产生了乱码:
nwnhqdif2#
从错误日志来看,问题出在模型配置文件(config.json)中缺少了一些必要的键值对。具体来说,
max_position_embeddings
、n_positions
、max_seq_len
、seq_length
等键值对没有在模型的配置文件中找到。这导致了模型的最大长度无法正确确定,从而引发了警告和错误。为了解决这个问题,你需要检查并修改模型的配置文件(config.json),确保其中包含了所有必要的键值对。你可以参考以下示例:
在这个示例中,我添加了
"quantization": "gptq"
键值对。你可以根据实际情况修改其他键值对。修改完成后,重新运行程序,问题应该可以得到解决。[rank0]: self.model_runner.load_model()
[rank0]: File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 134, in load_model
[rank0]: self.model = get_model(
[rank0]: File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/init.py", line 21, in get_model
[rank0]: return loader.load_model(model_config=model_config,
[rank0]: File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/loader.py", line 240, in load_model
[rank0]: model = _initialize_model(model_config, self.load_config,
[rank0]: File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/loader.py", line 91, in _initialize_model
[rank0]: return model_class(config=model_config.hf_config,
[rank0]: File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/falcon.py", line 389, in init
[rank0]: self.transformer = FalconModel(config, cache_config, quant_config)
[rank0]: File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/falcon.py", line 350, in init
[rank0]: self.h = nn.ModuleList([
[rank0]: File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/falcon.py", line 351, in
[rank0]: FalconDecoderLayer(config, cache_config, quant_config)
[rank0]: File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/falcon.py", line 249, in init
[rank0]: if (config.num_ln_in_parallel_attn is None
[rank0]: File "/home/ubuntu/.local/lib/python3.10/site-packages/transformers/configuration_utils.py", line 264, in getattribute
[rank0]: return super().getattribute(key)
[rank0]: AttributeError: 'RWConfig' object has no attribute 'num_ln_in_parallel_attn'
################################
I I changed the modifications here: /vllm/vllm/transformers_utils/configs$ falcon.py
self.bias = bias
self.parallel_attn = parallel_attn
self.new_decoder_architecture = new_decoder_architecture
self.num_ln_in_parallel_attn = None