text-generation-inference PaliGemma检测任务失败,

xwbd5t1u 于 22天前发布在其他

关注(0)|答案(1)|浏览(22)

系统信息

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100-SXM4-40GB          Off | 00000000:00:04.0 Off |                    0 |
| N/A   33C    P0              49W / 400W |  34683MiB / 40960MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      7589      G   /usr/lib/xorg/Xorg                           95MiB |
|    0   N/A  N/A      7836      G   /usr/bin/gnome-shell                         12MiB |
|    0   N/A  N/A     43844      C   /opt/conda/bin/python3.10                 34552MiB |
+---------------------------------------------------------------------------------------+

信息

Docker
直接使用CLI

任务

一个官方支持的命令
我自己的修改

复现

我正在使用Docker运行Google的PaliGemma 448-res模型：

model=google/paligemma-3b-pt-448
volume=$PWD/data
docker run --gpus all --shm-size 1g -e HF_TOKEN=$token -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:2.2.0 --model-id $model

我尝试使用这里的示例代码运行检测任务：

from huggingface_hub import InferenceClient

client = InferenceClient("http://127.0.0.1:8080")
image = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/rabbit.png"
prompt = f"![]({image})detect rabbit\n\n"
for token in client.text_generation(prompt, max_new_tokens=16, stream=True):
    print(token)

# This is a picture of an anthropomorphic rabbit in a space suit.

但我只得到了 <eos> 作为输出。
我在vLLM中提出了类似的问题： vllm-project/vllm#7115
我怀疑这与特殊的 <loc> 标记有关。在HF实现中，额外的位置标记被添加到Gemma处理器中：
https://github.com/huggingface/transformers/blob/25245ec26dc29bcf6102e1b4ddd0dfd02e720cf5/src/transformers/models/paligemma/processing_paligemma.py#L38
我不确定是否可能是这样，尽管如此。