sql with chunksize给出mysql数据的参数错误

5sxhfpxr  于 2021-06-24  发布在  Mysql
关注(0)|答案(1)|浏览(437)

我正在尝试将一个大型数据集(1300万行)从mysql数据库读入pandas(0.17.1)。根据网上的一个建议,我使用了 chunksize 参数来执行此操作。

  1. db = pymysql.connect(HOST, # localhost
  2. port=PORT, # port
  3. user=USER, # username
  4. password=PASSW, # password
  5. db=DATABASE) # name of the data base
  6. df = pd.DataFrame()
  7. query = "SELECT * FROM `table`;"
  8. for chunks in pd.read_sql(query, con=db, chunksize=100000):
  9. df = df.append(chunks)

但每次我做这件事都会有麻烦 TypeError: Argument 'rows' has incorrect type (expected list, got tuple) 错误。
当我没有使用chunksize参数,因此没有生成generator对象时,这就起作用了。我可以看到mysql正在返回一个 tuple-of-tuples 而不是 list-of-tuples .
所以,我的问题是,为什么查询在正常情况下工作,我该怎么做才能确保我从数据库中得到一个元组列表,这样我就可以使用它了?
完整的回溯看起来像这样

  1. TypeError Traceback (most recent call last)
  2. <ipython-input-20-efe94dcd2c70> in <module>()
  3. 8 df_horses = pd.DataFrame()
  4. 9 query = "SELECT * FROM `horses`;"
  5. ---> 10 for chunks in pd.read_sql(query, con=db, chunksize=10000):
  6. 11 df_horses = df_horses.append(chunks)
  7. 12 print df_horses.shape
  8. /home/ubuntu/anaconda2/lib/python2.7/site-packages/pandas/io/sql.pyc in _query_iterator(cursor, chunksize, columns, index_col, coerce_float, parse_dates)
  9. 1563 yield _wrap_result(data, columns, index_col=index_col,
  10. 1564 coerce_float=coerce_float,
  11. -> 1565 parse_dates=parse_dates)
  12. 1566
  13. 1567 def read_query(self, sql, index_col=None, coerce_float=True, params=None,
  14. /home/ubuntu/anaconda2/lib/python2.7/site-packages/pandas/io/sql.pyc in _wrap_result(data, columns, index_col, coerce_float, parse_dates)
  15. 135
  16. 136 frame = DataFrame.from_records(data, columns=columns,
  17. --> 137 coerce_float=coerce_float)
  18. 138
  19. 139 _parse_date_columns(frame, parse_dates)
  20. /home/ubuntu/anaconda2/lib/python2.7/site-packages/pandas/core/frame.pyc in from_records(cls, data, index, exclude, columns, coerce_float, nrows)
  21. 967 else:
  22. 968 arrays, arr_columns = _to_arrays(data, columns,
  23. --> 969 coerce_float=coerce_float)
  24. 970
  25. 971 arr_columns = _ensure_index(arr_columns)
  26. /home/ubuntu/anaconda2/lib/python2.7/site-packages/pandas/core/frame.pyc in _to_arrays(data, columns, coerce_float, dtype)
  27. 5277 if isinstance(data[0], (list, tuple)):
  28. 5278 return _list_to_arrays(data, columns, coerce_float=coerce_float,
  29. -> 5279 dtype=dtype)
  30. 5280 elif isinstance(data[0], collections.Mapping):
  31. 5281 return _list_of_dict_to_arrays(data, columns,
  32. /home/ubuntu/anaconda2/lib/python2.7/site-packages/pandas/core/frame.pyc in _list_to_arrays(data, columns, coerce_float, dtype)
  33. 5355 def _list_to_arrays(data, columns, coerce_float=False, dtype=None):
  34. 5356 if len(data) > 0 and isinstance(data[0], tuple):
  35. -> 5357 content = list(lib.to_object_array_tuples(data).T)
  36. 5358 else:
  37. 5359 # list of lists
  38. TypeError: Argument 'rows' has incorrect type (expected list, got tuple)
o8x7eapl

o8x7eapl1#

我不知道“pd.read\u sql”在使用chunksize时不返回元组列表的原因。事实上,“pd.read\u sql”不会引发版本“0.23.4”的任何错误。但我也尝试了Pandas版本“0.16.2”,在那里我遇到了和你一样的错误。所以请在编写脚本之前检查您的Pandas版本。但我知道一种方法可以克服Pandas版本“0.16.2”中的错误。

Pandas版本0.16.2

  1. import pymysql as ps
  2. import pandas as pd
  3. db=ps.connect(user="user_name", passwd="password", host = 'host_name',
  4. db='database_name')
  5. cursor=db.cursor()
  6. df=pd.DataFrame(columns=['column_name1','column_name2'])
  7. query=""" select column_name1,column_name2 from table_name limit {0},{1}; """
  8. limit=1000000
  9. offset=0
  10. try:
  11. while True:
  12. cursor.execute(query.format(offset,limit))
  13. rows=pd.DataFrame(list(cursor.fetchall()),columns=
  14. ['column_name1','column_name2'])
  15. df=pd.concat([df,rows],ignore_index=True)
  16. offset=offset+limit
  17. if len(rows['column_name1'])==0:
  18. break
  19. except:
  20. pass
展开查看全部

相关问题