当前环境
PyTorch version: 2.2.1+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.29.2
Libc version: glibc-2.35
Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-148-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA RTX A6000
Nvidia driver version: 525.105.17
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
Versions of relevant libraries:
[pip3] numpy==1.26.3
[pip3] nvidia-nccl-cu12==2.19.3
[pip3] torch==2.2.1
[pip3] torchaudio==2.2.0
[pip3] torchvision==0.17.0
[pip3] triton==2.2.0
[pip3] vllm_nccl_cu12==2.18.1.0.4.0
[conda] Could not collectROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.4.1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 CPU Affinity NUMA Affinity
GPU0 X 4-7,36-39 1
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
🐛 描述问题
我还没有完全隔离出这个问题。以下是我遇到的情况。
我将一部分GPU服务器从0.4.0-post1升级到了vLLM的0.4.1版本。升级后,我注意到与升级后的vLLM示例交互时,我的应用程序(不是来自vLLM本身)的错误率急剧上升。
我通过AsyncLLMEngine
类使用vLLM,特别是generate
方法。
错误是由vLLM生成部分特殊标记引起的。有问题的模型(https://huggingface.co/dreamgen/opus-v1-34b-awq)添加了特殊标记<|im_start|>
和<|im_end|>
。它应该始终将这些标记作为一组进行标记和生成。但在升级后,它以顺序生成这些块:<
<|
<|i
<|im
等。
降级解决了这个问题。
请注意,输入上的特殊标记被正确地标记,因此不太可能是标记器问题。
1条答案
按热度按时间1mrurvl11#
我也测试了0.4.2版本,不幸的是bug仍然存在。我想知道是否有可能进行发布回归测试,比较几个模型的输出logits与上一个已知的良好版本。