Now I am ready to make Xinference into an extensible image based on bge-reranker-large. How can I define the same model_id at startup and call it remotely through load balancing to create a pure GPU service like triton
Now I am ready to make Xinference into an extensible image based on bge-reranker-large. How can I define the same model_id at startup and call it remotely through load balancing to create a pure GPU service like triton
5条答案
按热度按时间zzwlnbp81#
你好,@Loovelj 。在启动模型时,你可以指定
model_uid
。bz4sfanl2#
你好,@Loovelj。你在启动模型时遇到了错误,可以尝试以下方法解决:
如果问题仍然存在,请查看日志以获取更多详细信息,并尝试重新启动模型。
1sbrub3j3#
Hi,@Loovelj。在启动模型时,您可以指定参数
model_uid
。谢谢,我已经知道了,但是这个有错误异常,能帮我解决吗?
`xinference launch --model-name bge-reranker-large --model-type rerank --model-uid my_model 2023-12-13 20:27:00,171 - modelscope - INFO - PyTorch version 2.1.1 Found. 2023-12-13 20:27:00,173 - modelscope - INFO - Loading ast index from /root/.cache/modelscope/ast_indexer 2023-12-13 20:27:02,440 - modelscope - INFO - Loading done! Current index file version is 1.10.0, with md5 4b2f06948d8e1e0227d48712befb1bbb and a total number of 946 components indexed Traceback (most recent call last): File "/root/anaconda3/envs/bge/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn conn = connection.create_connection( File "/root/anaconda3/envs/bge/lib/python3.9/site-packages/urllib3/util/connection.py", line 95, in create_connection raise err File "/root/anaconda3/envs/bge/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/root/anaconda3/envs/bge/lib/python3.9/site-packages/urllib3/connectionpool.py", line 715, in urlopen httplib_response = self._make_request( File "/root/anaconda3/envs/bge/lib/python3.9/site-packages/urllib3/connectionpool.py", line 416, in _make_request conn.request(method, url, **httplib_request_kw) File "/root/anaconda3/envs/bge/lib/python3.9/http/client.py", line 1285, in request super(HTTPConnection, self).request(method, url, body=body, headers=headers) File "/root/anaconda3/envs/bge/lib/python3.9/http/client.py", line 1331, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/root/anaconda3/envs/bge/lib/python3.9/http/client.py", line 1280, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/root/anaconda3/envs/bge/lib/python3.9/http/client.py", line 1040, in _send_output self.send(msg) File "/root/anaconda3/envs/bge/lib/python3.9/http/client.py", line 980, in send self.connect() File "/root/anaconda3/envs/bge
9gm1akwq4#
从错误日志来看,问题出在连接被拒绝。请检查以下几点:
model_uid
作为模型ID来启动模型。-e
选项指定Xinference端点。如果问题仍然存在,请提供更多关于您的环境和配置的信息,以便我们能更好地帮助您解决问题。
klr1opcd5#
Hi,@Loovelj