vllm [用法]:如何使用Medusa推测性采样推理模型,

jhkqcmku  于 5个月前  发布在  其他
关注(0)|答案(1)|浏览(52)

当前环境信息如下：

  • PyTorch版本：2.3.1+cu121
  • 是否为调试构建：否
  • 用于构建PyTorch的CUDA版本：12.1
  • 是否使用ROCM构建：N/A
  • 操作系统：Ubuntu 22.04.4 LTS(x86_64)
  • GCC版本：(Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
  • Clang版本：无法收集
  • CMake版本：3.30.0
  • Libc版本：glibc-2.35
  • Python版本：3.10.12(main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (64-bit)
  • Python平台：Linux-5.15.0-125.006-nvidia-x86_64-with-glibc2.35
  • 是否可用CUDA:是
  • CUDA运行时版本：无法收集
  • CUDA模块加载设置：LAZY
  • GPU模型和配置：
    GPU 0:NVIDIA H100 80GB HBM3
    GPU 1:NVIDIA H100 80GB HBM3
    GPU 2:NVIDIA H100 80GB HBM3
    GPU 3:NVIDIA H100 80GB HBM3
    GPU 4:NVIDIA H100 80GB HBM3
    GPU 5:NVIDIA H100 80GB HBM3
    GPU 6:NVIDIA H100 80GB HBM3
    GPU 7:NVIDIA H100 80GB HBM3
  • Nvidia驱动版本：550.90.07
  • cuDNN版本：无法收集
  • HIP运行时版本：N/A
  • MIOpen运行时版本：N/A
  • XNNPACK可用性：是

NIC4 节点

NODE NODE NODE NODE SYS SYS SYS SYS NODE NODE SYS SYS X

图例：

X = 自
SYS = 遍历 PCIe 以及在 NUMA 节点之间的 SMP 互连(例如，QPI/UPI)
NODE = 遍历 PCIe 以及在 NUMA 节点内的 PCIe 主机桥之间的互连
PHB = 遍历 PCIe 以及一个 PCIe 主机桥(通常是 CPU)
PXB = 遍历多个 PCIe 桥(不遍历 PCIe 主机桥)
PIX = 最多遍历一个 PCIe 桥
NV# = 遍历一组 # NVLinks 的绑定

NIC 图例：

NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_4
NIC3: mlx5_5
NIC4: mlx5_bond_0

如何使用 vllm

我想对 medusa model 进行推理。

启动服务器命令

  1. git clone https://huggingface.co/FasterDecoding/medusa-1.0-zephyr-7b-beta
  2. 从模型中提取 medausa_head weight 并将其设置为 medusa-1.0-zephyr-7b-beta/head
python3 -m vllm.entrypoints.openai.api_server --port 8010 \
  --model medusa-1.0-zephyr-7b-beta --dtype auto -tp 1 \
  --max-model-len 4096 --max-num-seqs 512 --gpu-memory-utilization 0.8 \
  --speculative-model medusa-1.0-zephyr-7b-beta/head \
  --speculative-draft-tensor-parallel-size 1 \
  --num-speculative-tokens 3 --speculative-disable-by-batch-size 4 \
  --use-v2-block-manager \
  --spec-decoding-acceptance-method typical_acceptance_sampler

问题：当使用 medausa 推测性采样时，性能较差(仅为基线性能的一半)。

INFO 07-24 04:00:41 metrics.py:295] Avg prompt throughput: 277.1 tokens/s, Avg generation throughput: 115.8 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
INFO 07-24 04:00:41 metrics.py:316] Speculative metrics: Draft acceptance rate: 0.000, System efficiency: 0.059, Number of speculative tokens: 16, Number of accepted tokens: 0, Number of draft tokens tokens: 60288, Number of emitted tokens tokens: 3768.

vllm 日志

INFO 07-24 03:57:14 api_server.py:212] vLLM API server version 0.5.2
INFO 07-24 03:57:14 api_server.py:213] args: Namespace(host=None, port=8005, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], model='/home/xxxx/models/medusa-1.0-zephyr-7b-beta', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, download_dir=None, load_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=4096, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, gpu_memory_utilization=0.8, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=128, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=False, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', scheduler_delay_factor=0.0, enable_chunked_prefill=False, speculative_model='/home/xxxx/models/medusa-1.0-zephyr-7b-beta/medusa', num_speculative_tokens=16, speculative_draft_tensor_parallel_size=1, speculative_max_model_len=None, speculative_disable_by_batch_size=4, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='typical_acceptance_sampler', typical_acceptance_sampler_posterior_threshold=0.01, typical_acceptance_sampler_posterior_alpha=0.1, model_loader_extra_config=None, preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, engine_use_ray=False, disable_log_requests=True, max_log_len=None)
INFO 07-24 03:57:14 config.py:1374] Downcasting torch.float32 to torch.float16.
INFO 07-24 03:57:14 llm_engine.py:174] Initializing an LLM engine (v0.5.2) with config: model='/home/xxxx/models/medusa-1.0-zephyr-7b-beta', speculative_config=SpeculativeConfig(draft_model='/home/xxxx/models/medusa-1.0-zephyr-7b-beta/medusa', num_spec_tokens=16), tokenizer='/home/xxxx/models/medusa-1.0-zephyr-7b-beta', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=4096, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=/home/xxxx/models/medusa-1.0-zephyr-7b-beta, use_v2_block_manager=True, enable_prefix_caching=False)
INFO 07-24 03:57:17 selector.py:169] Cannot use FlashAttention-2 backend due to sliding window.
INFO 07-24 03:57:17 selector.py:53] Using XFormers backend.
INFO 07-24 03:57:18 spec_decode_worker.py:141] Configuring SpecDecodeWorker with proposer=<class 'vllm.spec_decode.medusa_worker.MedusaWorker'>
INFO 07-24 03:57:18 spec_decode_worker.py:155] Configuring SpecDecodeWorker with sampler=<class 'vllm.model_executor.layers.typical_acceptance_sampler.TypicalAcceptanceSampler'>
INFO 07-24 03:57:19 selector.py:169] Cannot use FlashAttention-2 backend due to sliding window.
INFO 07-24 03:57:19 selector.py:53] Using XFormers backend.
INFO 07-24 03:57:31 model_runner.py:266] Loading model weights took 13.4966 GB
INFO 07-24 03:57:33 model_runner.py:266] Loading model weights took 4.4062 GB
INFO 07-24 03:57:34 gpu_executor.py:86] # GPU blocks: 22839, # CPU blocks: 2048
INFO 07-24 03:57:37 model_runner.py:1007] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
INFO 07-24 03:57:37 model_runner.py:1011] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
INFO 07-24 03:57:44 model_runner.py:1208] Graph capturing finished in 7 secs.
INFO 07-24 03:57:45 serving_chat.py:94] Using default chat template:^M
INFO 07-24 03:57:45 serving_chat.py:94] {% for message in messages %}^M
INFO 07-24 03:57:45 serving_chat.py:94] {% if message['role'] == 'user' %}^M
INFO 07-24 03:57:45 serving_chat.py:94] {{ '<|user|>^M
INFO 07-24 03:57:45 serving_chat.py:94] ' + message['content'] + eos_token }}^M
INFO 07-24 03:57:45 serving_chat.py:94] {% elif message['role'] == 'system' %}^M
INFO 07-24 03:57:45 serving_chat.py:94] {{ '<|system|>^M
INFO 07-24 03:57:45 serving_chat.py:94] ' + message['content'] + eos_token }}^M
INFO 07-24 03:57:45 serving_chat.py:94] {% elif message['role'] == 'assistant' %}^M
INFO 07-24 03:57:45 serving_chat.py:94] {{ '<|assistant|>^M
INFO 07-24 03:57:45 serving_chat.py:94] '  + message['content'] + eos_token }}^M
INFO 07-24 03:57:45 serving_chat.py:94] {% endif %}^M
INFO 07-24 03:57:45 serving_chat.py:94] {% if loop.last and add_generation_prompt %}^M
INFO 07-24 03:57:45 serving_chat.py:94] {{ '<|assistant|>' }}^M
INFO 07-24 03:57:45 serving_chat.py:94] {% endif %}^M
WARNING 07-24 03:57:45 serving_embedding.py:141] embedding_mode is False. Embedding API will not work.
INFO 07-24 03:57:45 api_server.py:257] Available routes are:
INFO 07-24 03:57:45 api_server.py:262] Route: /openapi.json, Methods: HEAD, GET
INFO 07-24 03:57:45 api_server.py:262] Route: /docs, Methods: HEAD, GET
INFO 07-24 03:57:45 api_server.py:262] Route: /docs/oauth2-redirect, Methods: HEAD, GET
INFO 07-24 03:57:45 api_server.py:262] Route: /redoc, Methods: HEAD, GET
INFO 07-24 03:57:45 api_server.py:262] Route: /health, Methods: GET
INFO 07-24 03:57:45 api_server.py:262] Route: /tokenize, Methods: POST
INFO 07-24 03:57:45 api_server.py:262] Route: /detokenize, Methods: POST
INFO 07-24 03:57:45 api_server.py:262] Route: /v1/models, Methods: GET
INFO 07-24 03:57:45 api_server.py:262] Route: /version, Methods: GET
INFO 07-24 03:57:45 api_server.py:262] Route: /v1/chat/completions, Methods: POST
INFO 07-24 03:57:45 api_server.py:262] Route: /v1/completions, Methods: POST
INFO 07-24 03:57:45 api_server.py:262] Route: /v1/embeddings, Methods: POST
INFO:     Started server process [37495]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8005 (Press CTRL+C to quit)
INFO 07-24 04:00:16 metrics.py:295] Avg prompt throughput: 4.0 tokens/s, Avg generation throughput: 135.3 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
INFO:     ::1:57296 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     ::1:57298 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     ::1:50174 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     ::1:50190 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     ::1:50198 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     ::1:50214 - "POST /v1/completions HTTP/1.1" 200 OK
INFO 07-24 04:00:21 metrics.py:295] Avg prompt throughput: 6.0 tokens/s, Avg generation throughput: 137.5 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 07-24 04:00:21 metrics.py:316] Speculative metrics: Draft acceptance rate: 0.000, System efficiency: 0.059, Number of speculative tokens: 16, Number of accepted tokens: 0, Number of draft tokens tokens: 21664, Number of emitted tokens tokens: 1354.
INFO:     ::1:50216 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     ::1:50230 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     ::1:50236 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     ::1:50242 - "POST /v1/completions HTTP/1.1" 200 OK
INFO 07-24 04:00:26 metrics.py:295] Avg prompt throughput: 4.0 tokens/s, Avg generation throughput: 131.9 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
INFO 07-24 04:00:26 metrics.py:316] Speculative metrics: Draft acceptance rate: 0.000, System efficiency: 0.059, Number of speculative tokens: 16, Number of accepted tokens: 0, Number of draft tokens tokens: 32160, Number of emitted tokens tokens: 2010.
INFO:     ::1:50258 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     ::1:50266 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     ::1:50328 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     ::1:50332 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     ::1:50338 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     ::1:50342 - "POST /v1/completions HTTP/1.1" 200 OK
INFO 07-24 04:00:31 metrics.py:295] Avg prompt throughput: 558.7 tokens/s, Avg generation throughput: 127.4 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.8%, CPU KV cache usage: 0.0%.
INFO 07-24 04:00:31 metrics.py:316] Speculative metrics: Draft acceptance rate: 0.000, System efficiency: 0.059, Number of speculative tokens: 16, Number of accepted tokens: 0, Number of draft tokens tokens: 42272, Number of emitted tokens tokens: 2642.
INFO:     ::1:50352 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     ::1:50356 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     ::1:50368 - "POST /v1/completions HTTP/1.1" 200 OK
INFO 07-24 04:00:36 metrics.py:295] Avg prompt throughput: 278.5 tokens/s, Avg generation throughput: 110.4 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
INFO 07-24 04:00:36 metrics.py:316] Speculative metrics: Draft acceptance rate: 0.000, System efficiency: 0.059, Number of speculative tokens: 16, Number of accepted tokens: 0, Number of draft tokens tokens: 51056, Number of emitted tokens tokens: 3191.
INFO:     ::1:50382 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     ::1:40776 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     ::1:40792 - "POST /v1/completions HTTP/1.1" 200 OK
INFO 07-24 04:00:41 metrics.py:295] Avg prompt throughput: 277.1 tokens/s, Avg generation throughput: 115.8 tokens/s, Running: 2 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.4%, CPU KV cache usage: 0.0%.
INFO 07-24 04:00:41 metrics.py:316] Speculative metrics: Draft acceptance rate: 0.000, System efficiency: 0.059, Number of speculative tokens: 16, Number of accepted tokens: 0, Number of draft tokens tokens: 60288, Number of emitted tokens tokens: 3768.
INFO:     ::1:40804 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     ::1:40816 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     ::1:40826 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     ::1:40836 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     ::1:57242 - "GET /v1/models HTTP/1.1" 200 OK
vllm
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com