系统信息

硬件:AWS g6.12xlarge (us-east-2) / 4x NVIDIA L4 GPU
操作系统:Ubuntu 24.04 LTS (Noble Numbat)
NVIDIA驱动:nvidia-open 560.28.03
CUDA:12.6
Docker:Docker版本27.1.1,构建6312585
NVIDIA容器工具包:1.16.1
TGI:ghcr.io/huggingface/text-generation-inference:latest(4b44be4c038f)

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.28.03              Driver Version: 560.28.03      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L4                      Off |   00000000:38:00.0 Off |                    0 |
| N/A   41C    P8             16W /   72W |       1MiB /  23034MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA L4                      Off |   00000000:3A:00.0 Off |                    0 |
| N/A   42C    P8             17W /   72W |       1MiB /  23034MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA L4                      Off |   00000000:3C:00.0 Off |                    0 |
| N/A   41C    P8             17W /   72W |       1MiB /  23034MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA L4                      Off |   00000000:3E:00.0 Off |                    0 |
| N/A   38C    P8             16W /   72W |       1MiB /  23034MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

信息

Docker
直接使用CLI

任务

一个官方支持的命令
我自己的修改

重现

要重现，请运行以下shell脚本：

model=neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8
volume=$PWD/weights
token=<REDACTED>
docker run --rm --runtime nvidia --gpus all --shm-size 1g -p 8080:80 -v $volume:/data \
    ghcr.io/huggingface/text-generation-inference:latest --model-id $model

启动过程中出现以下异常：

2024-08-09T12:44:30.578630Z  INFO text_generation_launcher: Args {
    model_id: "neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8",
    revision: None,
    validation_workers: 2,
    sharded: None,
    num_shard: None,
    quantize: None,
    speculate: None,
    dtype: None,
    trust_remote_code: false,
    max_concurrent_requests: 128,
    max_best_of: 2,
    max_stop_sequences: 4,
    max_top_n_tokens: 5,
    max_input_tokens: None,
    max_input_length: None,
    max_total_tokens: None,
    waiting_served_ratio: 0.3,
    max_batch_prefill_tokens: None,
    max_batch_total_tokens: None,
    max_waiting_tokens: 20,
    max_batch_size: None,
    cuda_graphs: None,
    port: 80,
    shard_uds_path: "/tmp/text-generation-server",
    master_addr: "localhost",
    master_port: 29500,
    huggingface_hub_cache: Some(
        "/data",
    ),
    weights_cache_override: None,
    disable_custom_kernels: false,
    cuda_memory_fraction: 1.0,
    rope_scaling: None,
    rope_factor: None,
    json_output: false,
    otlp_endpoint: None,
    otlp_service_name: "text-generation-inference.router",
    cors_allow_origin: [],
    api_key: None,
    watermark_gamma: None,
    watermark_delta: None,
    ngrok: false,
    ngrok_authtoken: None,
    ngrok_edge: None,
    tokenizer_config_path: None,
    disable_grammar_support: false,
    env: false,
    max_client_batch_size: 4,
    lora_adapters: None,
    usage_stats: On,
}
2024-08-09T12:44:30.578721Z  INFO hf_hub: Token file not found "/root/.cache/huggingface/token"
2024-08-09T12:44:30.632826Z  INFO text_generation_launcher: Model supports up to 131072 but tgi will now set its default to 4096 instead. This is to save VRAM by refusing large prompts in order to allow more users on the same hardware. You can increase that size using `--max-batch-prefill-tokens=131122 --max-total-tokens=131072 --max-input-tokens=131071`.
2024-08-09T12:44:30.632839Z  INFO text_generation_launcher: Default `max_input_tokens` to 4095
2024-08-09T12:44:30.632841Z  INFO text_generation_launcher: Default `max_total_tokens` to 4096
2024-08-09T12:44:30.632843Z  INFO text_generation_launcher: Default `max_batch_prefill_tokens` to 4145
2024-08-09T12:44:30.632844Z  INFO text_generation_launcher: Using default cuda graphs [1, 2, 4, 8, 16, 32]
2024-08-09T12:44:30.632939Z  INFO download: text_generation_launcher: Starting check and download process for neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8
2024-08-09T12:44:33.840860Z  INFO text_generation_launcher: Files are already present on the host. Skipping download.
2024-08-09T12:44:34.638228Z  INFO download: text_generation_launcher: Successfully downloaded weights for neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8
2024-08-09T12:44:34.638608Z  INFO shard-manager: text_generation_launcher: Starting shard rank=0
2024-08-09T12:44:39.179449Z  INFO text_generation_launcher: GPU does not support FP8, using Marlin FP8 kernel
2024-08-09T12:44:39.213012Z ERROR text_generation_launcher: Error when initializing model
Traceback (most recent call last):
  File "/opt/conda/bin/text-generation-server", line 8, in <module>
    sys.exit(app())
  File "/opt/conda/lib/python3.10/site-packages/typer/main.py", line 311, in __call__
    return get_command(self)(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
:Error: ShardCannotStart 
    return self.main(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/typer/core.py", line 778, in main
    return _main(
  File "/opt/conda/lib/python3.10/site-packages/typer/core.py", line 216, in _main
    rv = self.invoke(ctx)
  File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/typer/main.py", line 683, in wrapper
    return callback(**use_params)  # type: ignore
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/cli.py", line 109, in serve
    server.serve(
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/server.py", line 274, in serve
    asyncio.run(
  File "/opt/conda/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 636, in run_until_complete
    self.run_forever()
  File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
    self._run_once()
  File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
    handle._run()
  File "/opt/conda/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
> File "/opt/conda/lib/python3.10/site-packages/text_generation_server/server.py", line 229, in serve_inner
    model = get_model_with_lora_adapters(
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/__init__.py", line 1195, in get_model_with_lora_adapters
    model = get_model(
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/__init__.py", line 766, in get_model
    return FlashCausalLM(
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/flash_causal_lm.py", line 896, in __init__
    model = model_class(prefix, config, weights)
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_llama_modeling.py", line 528, in __init__
    self.model = FlashLlamaModel(prefix, config, weights)
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_llama_modeling.py", line 418, in __init__
    FlashLlamaLayer(
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_llama_modeling.py", line 346, in __init__
    self.self_attn = FlashLlamaAttention(
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_llama_modeling.py", line 166, in __init__
    self.query_key_value = load_attention(config, prefix, weights, index)
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_llama_modeling.py", line 94, in load_attention
    base_layer = TensorParallelColumnLinear.load_multi(
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/layers/tensor_parallel.py", line 179, in load_multi
    linear = get_linear(weight, bias)
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/layers/linear.py", line 102, in get_linear
    return weight.get_linear(bias)
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/layers/fp8.py", line 185, in get_linear
    return get_fp8_linear().from_fp8(
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/layers/marlin/fp8.py", line 66, in from_fp8
    return cls(qweight=weight, scales=scale.to(dtype), bias=bias)
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/layers/marlin/fp8.py", line 45, in __init__
    qweight, scales = repack_fp8_for_marlin(qweight, scales)
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/layers/marlin/fp8.py", line 138, in repack_fp8_for_marlin
    scales = permute_scales(scales)
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/layers/marlin/util.py", line 48, in permute_scales
    scales = scales.reshape((-1, len(scale_perm_single)))[:, scale_perm_single]
RuntimeError: shape '[-1, 32]' is invalid for input of size 3
2024-08-09T12:44:40.147176Z ERROR shard-manager: text_generation_launcher: Shard complete standard error output:
2024-08-09 12:44:36.570 | INFO     | text_generation_server.utils.import_utils:<module>:73 - Detected system cuda
/opt/conda/lib/python3.10/site-packages/text_generation_server/utils/sgmv.py:18: UserWarning: Could not import SGMV kernel from Punica, falling back to loop.
  warnings.warn("Could not import SGMV kernel from Punica, falling back to loop.")
/opt/conda/lib/python3.10/site-packages/mamba_ssm/ops/selective_scan_interface.py:159: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  def forward(ctx, xz, conv1d_weight, conv1d_bias, x_proj_weight, delta_proj_weight,
/opt/conda/lib/python3.10/site-packages/mamba_ssm/ops/selective_scan_interface.py:232: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., device_type='cuda')` instead.
  def backward(ctx, dout):
/opt/conda/lib/python3.10/site-packages/mamba_ssm/ops/triton/layernorm.py:508: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
  def forward(
/opt/conda/lib/python3.10/site-packages/mamba_ssm/ops/triton/layernorm.py:567: FutureWarning: `torch.cuda.amp.custom_bwd(args...)` is deprecated. Please use `torch.amp.custom_bwd(args..., device_type='cuda')` instead.
  def backward(ctx, dout, *args):
/opt/conda/lib/python3.10/site-packages/torch/distributed/c10d_logger.py:79: FutureWarning: You are using a Backend <class 'text_generation_server.utils.dist.FakeGroup'> as a ProcessGroup. This usage is deprecated since PyTorch 2.0. Please use a public API of PyTorch Distributed instead.
  return func(*args, **kwargs)
Traceback (most recent call last):
  File "/opt/conda/bin/text-generation-server", line 8, in <module>
    sys.exit(app())
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/cli.py", line 109, in serve
    server.serve(
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/server.py", line 274, in serve
    asyncio.run(
  File "/opt/conda/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/server.py", line 229, in serve_inner
    model = get_model_with_lora_adapters(
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/__init__.py", line 1195, in get_model_with_lora_adapters
    model = get_model(
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/__init__.py", line 766, in get_model
    return FlashCausalLM(
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/flash_causal_lm.py", line 896, in __init__
    model = model_class(prefix, config, weights)
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_llama_modeling.py", line 528, in __init__
    self.model = FlashLlamaModel(prefix, config, weights)
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_llama_modeling.py", line 418, in __init__
    FlashLlamaLayer(
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_llama_modeling.py", line 346, in __init__
    self.self_attn = FlashLlamaAttention(
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_llama_modeling.py", line 166, in __init__
    self.query_key_value = load_attention(config, prefix, weights, index)
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/custom_modeling/flash_llama_modeling.py", line 94, in load_attention
    base_layer = TensorParallelColumnLinear.load_multi(
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/layers/tensor_parallel.py", line 179, in load_multi
    linear = get_linear(weight, bias)
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/layers/linear.py", line 102, in get_linear
    return weight.get_linear(bias)
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/layers/fp8.py", line 185, in get_linear
    return get_fp8_linear().from_fp8(
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/layers/marlin/fp8.py", line 66, in from_fp8
    return cls(qweight=weight, scales=scale.to(dtype), bias=bias)
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/layers/marlin/fp8.py", line 45, in __init__
    qweight, scales = repack_fp8_for_marlin(qweight, scales)
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/layers/marlin/fp8.py", line 138, in repack_fp8_for_marlin
    scales = permute_scales(scales)
  File "/opt/conda/lib/python3.10/site-packages/text_generation_server/layers/marlin/util.py", line 48, in permute_scales
    scales = scales.reshape((-1, len(scale_perm_single)))[:, scale_perm_single]
RuntimeError: shape '[-1, 32]' is invalid for input of size 3
 rank=0
2024-08-09T12:44:40.244304Z ERROR text_generation_launcher: Shard 0 failed to start

预期行为

我希望在分片初始化期间不会出现任何问题。TGI应该能够正常启动并提供模型服务。

4条答案

按热度按时间

yhxst69z1#

虽然我在这里没有找到已经报告的问题，但我确实发现了一个与vLLM类似的问题。他们甚至刚刚修复了它，所以也许他们的更改可以移植到TGI?

vLLM问题：[Bug]: Dynamic FP8 Marlin quantization fails on 0.5.4 vllm-project/vllm#7216
vLLM PR:[Bugfix][FP8] Fix dynamic FP8 Marlin quantization vllm-project/vllm#7219

赞(0）回复(0）举报 7个月前

pexxcrt22#

你好，@DrNochi 👋
感谢你报告这个问题，我会@danieldk,他是一个马林Maven！

kmbjn2e33#

附注：为什么TGI甚至要回退到使用Marlin内核？据我所知，NVIDIA L4正在使用具有CUDA计算能力8.9的艾达·洛夫莱斯架构，这应该为FP提供硬件支持：NVIDIA CUDA文档。我是不是漏掉了什么？
快速浏览了一下代码，我发现了以下PR #2277,这是最新版本的一部分，基本上“阻止”TGI利用原生FP8支持，通过强制Marlin内核用于CC 8.9。我没有发现任何与这些更改相关的问题或进一步解释。也许@OlivierDehaene可以解释一下这个改变背后的原因？

3vpjnl9f4#

附注：为什么TGI甚至开始使用Marlin内核？据我所知，NVIDIA L4正在使用具有CUDA计算能力8.9的Ada Lovelace架构，这应该为FP提供硬件支持：NVIDIA CUDA文档。我是不是漏掉了什么？
我们切换到了fbgemm-gpu进行FP8矩阵乘法。然而，它使用了TMA(Tensor内存加速器),而在CC 8.9中并不受支持。

text-generation-inference [BUG] 在NVIDIA L4上运行FP8量化模型失败(repack_fp8_for_marlin)

系统信息

信息

任务

重现

预期行为

4条答案

相关问题

热门标签

最新问答