系统信息

tgi docker镜像 2.0.4

信息

• Docker
• CLI直接使用

任务

• 一个官方支持的命令
• 我自己的修改

重现

将以下内容传递给聊天完成路由：

{
 "logprobs": true,
 "max_tokens": 1,
 "messages": [
 {
 "role": "user",
 "content": "什么是深度学习？"
 }
 ],
 "model": "mistralai/Mistral-7B-Instruct-v0.2",
 "top_logprobs": 3,
 "top_p": 0.5
}

得到：

{
 "id": "",
 "object": "text_completion",
 "created": 1719230167,
 "model": "/data/models--meta-llama--Meta-Llama-3-8B-Instruct/snapshots/e1945c40cd546c78e41f1151f4db032b271faeaa",
 "system_fingerprint": "2.0.4-sha-f426a33",
 "choices": [
 {
 "index": 0,
 "message": {
 "role": "assistant",
 "content": "Deep"
 },
 "logprobs": {
 "content": [
 {
 "token": "Deep",
 "logprob": 0,
 "top_logprobs": [
 {
 "token": "Deep",
 "logprob": 0
 }
 ]
 }
 ]
 },
 "finish_reason": "length"
 }
 ],
 "usage": {
 "prompt_tokens": 15,
 "completion_tokens": 1,
 "total_tokens": 16
 }
}

而当删除 top_p 时，输出如下(请查看 top_logprobs):

{
 "id": "",
 "object": "text_completion",
 "created": 1719230253,
 "model": "/data/models--meta-llama--Meta-Llama-3-8B-Instruct/snapshots/e1945c40cd546c78e41f1151f4db032b271faeaa",
 "system_fingerprint": "2.0.4-sha-f426a33",
 "choices": [
 {
 "index": 0,
 "message": {
 "role": "assistant",
 "content": "Deep"
 },
 "logprobs": {
 "content": [
 {
 "token": "Deep",
 "logprob": -0.005470276,
 "top_logprobs": [
 {
 "token": "Deep", **注意**:这里的值发生了变化**,应保持不变**,即 top_logprobs[0].logprob 应为 0。**,但实际上是-0.005470276**,这与预期不符。**,但实际上是-0.005470276**,这与预期不符。**,但实际上是-0.005470276**,这与预期不符。**,但实际上是-0.005470276**,这与预期不符。**,但实际上是-0.005470276**,这与预期不符。**,但实际上是-0.005470276**,这与预期不符。**,但实际上是-0.005470276**,这与预期不符。**,但实际上是-0.005470276**,这与预期不符。**,但实际上是-0.005470276**,这与预期不符。**,但实际上是-0.