在vLM和Xinference上部署相同的模型时,langchain可以连接到它们。vLM上的模型功能也可以正常使用,但是在Xinference上已经运行起来的模型与langchain之间存在问题。使用的Xinference版本是0.13.1,postman也有响应,但是langchain与xinference之间出现问题:
RemoteProtocolError: peer closed connection without sending complete message body (incomplete chunked read)
Traceback:
File "/home/user/miniconda3/envs/langchain3/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 600, in _run_script
exec(code, module.dict)
File "/home/user/Langchain-Chatchat-master_0_3_1/libs/chatchat-server/chatchat/webui.py", line 73, in dialogue_page(api=api, is_lite=is_lite)
File "/home/user/Langchain-Chatchat-master_0_3_1/libs/chatchat-server/chatchat/webui_pages/dialogue/dialogue.py", line 455, in dialogue_page
for d in client.chat.completions.create(
File "/home/user/miniconda3/envs/langchain3/lib/python3.10/site-packages/openai/_streaming.py", line 46, in iter
for item in self._iterator:
File "/home/user/miniconda3/envs/langchain3/lib/python3.10/site-packages/openai/_streaming.py", line 58, in stream
for sse in iterator:
File "/home/user/miniconda3/envs/langchain3/lib/python3.10/site-packages/openai/_streaming.py", line 50, in _iter_events
yield from self._decoder.iter_bytes(self.response.iter_bytes())
File "/home/user/miniconda3/envs/langchain3/lib/python3.10/site-packages/openai/_streaming.py", line 280, in iter_bytes
for chunk in self._iter_chunks(iterator):
File "/home/user/miniconda3/envs/langchain3/lib/python3.10/site-packages/openai/_streaming.py", line 291, in _iter_chunks
for chunk in iterator:
File "/home/user/miniconda3/envs/langchain3/lib/python3.10/site-packages/httpx/_models.py", line 829, in iter_bytes
for raw_bytes in self.iter_raw():
File "/home/user
1条答案
按热度按时间7bsow1i61#
您好,这个问题可能是由于模型引擎不兼容导致的。您可以尝试使用vllm替换原来的model engine,这样可能会解决问题。