python在长时间运行的hive insert查询期间“tsocket read 0 bytes”

oyt4ldly  于 2021-06-27  发布在  Hive
关注(0)|答案(1)|浏览(1440)

我正在使用pyhive0.6.1在hive中运行一个长的ish insert查询，结果失败了 thrift.transport.TTransport.TTransportException: TSocket read 0 bytes 大约5分钟后。在服务器端，查询将一直运行，直到成功完成。我对快速查询没有这个问题。
我不能用相同的python版本在mac上本地复制它：代码正确地等待，直到查询完成。发生这种情况的环境是基于python:3.6-slim. 除此之外，我正在安装libsasl2 dev和libsasl2 modules包，以及pyhive[hive]python包。
你知道为什么会这样吗？提前谢谢。
我使用的代码是：

  1. import contextlib
  2. from pyhive.hive import connect
  4. def get_conn():
  5.     return connect(
  6.         host='my-host',
  7.         port=10000,
  8.         auth='NONE',
  9.         username='username',
  10.         database='database'
  11.     )
  13. with contextlib.closing(get_conn()) as conn, \
  14.         contextlib.closing(conn.cursor()) as cur:
  15.     cur.execute('My long insert statement')

这是完整的回溯

  1. Traceback (most recent call last):
  2.   File "<stdin>", line 5, in <module>
  3.   File "/usr/local/lib/python3.6/site-packages/pyhive/hive.py", line 364, in execute
  4.     response = self._connection.client.ExecuteStatement(req)
  5.   File "/usr/local/lib/python3.6/site-packages/TCLIService/TCLIService.py", line 280, in ExecuteStatement
  6.     return self.recv_ExecuteStatement()
  7.   File "/usr/local/lib/python3.6/site-packages/TCLIService/TCLIService.py", line 292, in recv_ExecuteStatement
  8.     (fname, mtype, rseqid) = iprot.readMessageBegin()
  9.   File "/usr/local/lib/python3.6/site-packages/thrift/protocol/TBinaryProtocol.py", line 134, in readMessageBegin
  10.     sz = self.readI32()
  11.   File "/usr/local/lib/python3.6/site-packages/thrift/protocol/TBinaryProtocol.py", line 217, in readI32
  12.     buff = self.trans.readAll(4)
  13.   File "/usr/local/lib/python3.6/site-packages/thrift/transport/TTransport.py", line 60, in readAll
  14.     chunk = self.read(sz - have)
  15.   File "/usr/local/lib/python3.6/site-packages/thrift_sasl/__init__.py", line 166, in read
  16.     self._read_frame()
  17.   File "/usr/local/lib/python3.6/site-packages/thrift_sasl/__init__.py", line 170, in _read_frame
  18.     header = self._trans.readAll(4)
  19.   File "/usr/local/lib/python3.6/site-packages/thrift/transport/TTransport.py", line 60, in readAll
  20.     chunk = self.read(sz - have)
  21.   File "/usr/local/lib/python3.6/site-packages/thrift/transport/TSocket.py", line 132, in read
  22.     message='TSocket read 0 bytes')
  23. thrift.transport.TTransport.TTransportException: TSocket read 0 bytes
  25. During handling of the above exception, another exception occurred:
  27. Traceback (most recent call last):
  28.   File "<stdin>", line 5, in <module>
  29.   File "/usr/local/lib/python3.6/contextlib.py", line 185, in __exit__
  30.     self.thing.close()
  31.   File "/usr/local/lib/python3.6/site-packages/pyhive/hive.py", line 221, in close
  32.     response = self._client.CloseSession(req)
  33.   File "/usr/local/lib/python3.6/site-packages/TCLIService/TCLIService.py", line 218, in CloseSession
  34.     return self.recv_CloseSession()
  35.   File "/usr/local/lib/python3.6/site-packages/TCLIService/TCLIService.py", line 230, in recv_CloseSession
  36.     (fname, mtype, rseqid) = iprot.readMessageBegin()
  37.   File "/usr/local/lib/python3.6/site-packages/thrift/protocol/TBinaryProtocol.py", line 134, in readMessageBegin
  38.     sz = self.readI32()
  39.   File "/usr/local/lib/python3.6/site-packages/thrift/protocol/TBinaryProtocol.py", line 217, in readI32
  40.     buff = self.trans.readAll(4)
  41.   File "/usr/local/lib/python3.6/site-packages/thrift/transport/TTransport.py", line 60, in readAll
  42.     chunk = self.read(sz - have)
  43.   File "/usr/local/lib/python3.6/site-packages/thrift_sasl/__init__.py", line 166, in read
  44.     self._read_frame()
  45.   File "/usr/local/lib/python3.6/site-packages/thrift_sasl/__init__.py", line 170, in _read_frame
  46.     header = self._trans.readAll(4)
  47.   File "/usr/local/lib/python3.6/site-packages/thrift/transport/TTransport.py", line 60, in readAll
  48.     chunk = self.read(sz - have)
  49.   File "/usr/local/lib/python3.6/site-packages/thrift/transport/TSocket.py", line 132, in read
  50.     message='TSocket read 0 bytes')
  51. thrift.transport.TTransport.TTransportException: TSocket read 0 bytes
Hivepythonthriftpyhive

1条答案

按热度按时间
gopyfrb3

gopyfrb31#

引发的代码异常如下：

  1. def read(self, sz):
  2.         try:
  3.             buff = self.handle.recv(sz)
  4.         except socket.error as e:
  5.             if (e.args[0] == errno.ECONNRESET and
  6.                     (sys.platform == 'darwin' or sys.platform.startswith('freebsd'))):
  7.                 # freebsd and Mach don't follow POSIX semantic of recv
  8.                 # and fail with ECONNRESET if peer performed shutdown.
  9.                 # See corresponding comment and code in TSocket::read()
  10.                 # in lib/cpp/src/transport/TSocket.cpp.
  11.                 self.close()
  12.                 # Trigger the check to raise the END_OF_FILE exception below.
  13.                 buff = ''
  14.             else:
  15.                 raise
  16.         if len(buff) == 0:
  17.             raise TTransportException(type=TTransportException.END_OF_FILE,
  18.                                       message='TSocket read 0 bytes')
  19.         return buff

我想这可能是会议提出的 errno.ECONNRESET 服务器故意关闭连接时可能发生的错误(根据这个节点js econnreset，服务器可能会在太忙时终止连接，并发现连接已超过“keep alive”超时）
您可以检查服务器的日志，并找出是否存在某种终止连接行为。
我不确定，只是提供了我的想法。

展开查看全部
赞(0）分享  回复(0）举报  2021-06-27
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com