vllm [特性]:支持不使用nvcc的kv-cache-dtype fp8

nsc4cvqm 于 3个月前发布在其他

关注(0)|答案(4)|浏览(32)

🚀 功能、动机和宣传

目前，如果没有安装nvcc,会抛出以下错误：

WARNING 05-08 01:34:59 utils.py:313] Not found nvcc in /usr/local/cuda. Skip cuda version check!
Traceback (most recent call last):                                                                              
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,                                                                  
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)                                                                                     
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 159, in <module>
    engine = AsyncLLMEngine.from_engine_args(                                                                   
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 341, in from_engine_args
    engine_config = engine_args.create_engine_config()
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/arg_utils.py", line 471, in create_engine_config
    cache_config = CacheConfig(self.block_size,
  File "/usr/local/lib/python3.10/dist-packages/vllm/config.py", line 310, in __init__
    self._verify_cache_dtype()
  File "/usr/local/lib/python3.10/dist-packages/vllm/config.py", line 333, in _verify_cache_dtype
    if nvcc_cuda_version < Version("11.8"):
TypeError: '<' not supported between instances of 'NoneType' and 'Version'

然而，我们应该能够在不依赖于nvcc的情况下进行版本检查。所以让我们移除这个依赖关系。