pandas to_gbq错误:“pyarrow.lib.ArrowNotImplementedError:NumPyConverter不实现&lt;list< item: int64>&gt;转换”

cygmwpex  于 2023-06-28  发布在  其他
关注(0)|答案(1)|浏览(234)

我想从MySQL复制一个表到BigQuery。我有以下云函数:

import pandas as pd
from sqlalchemy import create_engine
import pandas_gbq

def query_mysql(request):
    engine = create_engine(
        "mysql+pymysql://bd:pass@host/user"
    )
    df = pd.read_sql("SELECT * FROM table", engine)

    df.info()
    df.head(10)

    pandas_gbq.to_gbq(
        df,
        destination,
        project_id,
        if_exists="replace"
    )

运行此函数时,出现以下错误"pyarrow.lib.ArrowNotImplementedError: NumPyConverter doesn't implement <list<item: int64>> conversion"
下面是.info()的输出:

DEFAULT 2023-06-18T17:58:36.791332Z Data columns (total 13 columns):
DEFAULT 2023-06-18T17:58:36.791337Z # Column Non-Null Count Dtype
DEFAULT 2023-06-18T17:58:36.791342Z --- ------ -------------- -----
DEFAULT 2023-06-18T17:58:36.791347Z 0 SKU 7771 non-null object
DEFAULT 2023-06-18T17:58:36.791352Z 1 text 7771 non-null object
DEFAULT 2023-06-18T17:58:36.791357Z 2 document_date1 7771 non-null datetime64[ns]
DEFAULT 2023-06-18T17:58:36.791361Z 3 Sum_quantity1 7771 non-null float64
DEFAULT 2023-06-18T17:58:36.791365Z 4 Landon 7771 non-null object
DEFAULT 2023-06-18T17:58:36.791370Z 5 Category 7771 non-null object
DEFAULT 2023-06-18T17:58:36.791374Z 6 Wheel 7771 non-null float64
DEFAULT 2023-06-18T17:58:36.791379Z 7 Frame 7771 non-null object
DEFAULT 2023-06-18T17:58:36.791383Z 8 Version 7771 non-null object
DEFAULT 2023-06-18T17:58:36.791387Z 9 Color 7771 non-null object
DEFAULT 2023-06-18T17:58:36.791391Z 10 YYYY_MM 7771 non-null object
DEFAULT 2023-06-18T17:58:36.791399Z 11 month 7771 non-null int64
DEFAULT 2023-06-18T17:58:36.791406Z 12 code_of_country 7771 non-null object

这是.head()

DEFAULT 2023-06-18T18:37:08.481079Z SKU text document_date1 \
DEFAULT 2023-06-18T18:37:08.481103Z 0 11-00-208 Veloped Tour 14er L black/reflex/red 2020-07-22
DEFAULT 2023-06-18T18:37:08.481109Z 1 11-00-103 Veloped Sport 12er M blk/wht/wht 2018-02-01
DEFAULT 2023-06-18T18:37:08.481114Z 2 11-00-209 Veloped Trek 14er M green/blk/ora 2022-02-17
DEFAULT 2023-06-18T18:37:08.481120Z 3 11-00-207 Veloped Tour 14er M svart/reflex/röd 2018-06-13
DEFAULT 2023-06-18T18:37:08.481124Z 4 24-00-001 Rollator Walker 12er schwarz/schwarz/grau 2020-01-21
DEFAULT 2023-06-18T18:37:08.481129Z ... ... ... ...
DEFAULT 2023-06-18T18:37:08.481134Z 7766 11-00-203 Veloped Sport 14er M sv/vit/vit 2021-03-04
DEFAULT 2023-06-18T18:37:08.481139Z 7767 11-00-107 Veloped Tour 12er M schwarz/reflex/rot 2020-09-30
DEFAULT 2023-06-18T18:37:08.481145Z 7768 11-00-207 Veloped Tour 14er M black/reflex/red 2021-11-17
DEFAULT 2023-06-18T18:37:08.481151Z 7769 11-00-209 Veloped Trek 14er M green/black/orange 2022-05-18
DEFAULT 2023-06-18T18:37:08.481156Z 7770 25-00-001 Walker 14er black/black/gray 2019-07-08
DEFAULT 2023-06-18T18:37:08.481163Z Sum_quantity1 Landon Category Wheel Frame Version Color \
DEFAULT 2023-06-18T18:37:08.481168Z 0 1.0 11-00-202 Veloped 14.0 L Tour Black
DEFAULT 2023-06-18T18:37:08.481172Z 1 3.0 11-00-101 Veloped 12.0 M Sport Black
DEFAULT 2023-06-18T18:37:08.481177Z 2 1.0 11-00-201 Veloped 14.0 M Trek Green
DEFAULT 2023-06-18T18:37:08.481181Z 3 1.0 11-00-201 Veloped 14.0 M Tour Black
DEFAULT 2023-06-18T18:37:08.481186Z 4 1.0 24-00-001 Walker 12.0 M/L Silver 12er Black
DEFAULT 2023-06-18T18:37:08.481191Z ... ... ... ... ... ... ... ...
DEFAULT 2023-06-18T18:37:08.481195Z 7766 1.0 11-00-201 Veloped 14.0 M Sport Black
DEFAULT 2023-06-18T18:37:08.481200Z 7767 1.0 11-00-101 Veloped 12.0 M Tour Black
DEFAULT 2023-06-18T18:37:08.481205Z 7768 1.0 11-00-201 Veloped 14.0 M Tour Black
DEFAULT 2023-06-18T18:37:08.481210Z 7769 1.0 11-00-201 Veloped 14.0 M Trek Green
DEFAULT 2023-06-18T18:37:08.481214Z 7770 1.0 25-00-001 Walker 14.0 L Black 14er Black
DEFAULT 2023-06-18T18:37:08.481220Z YYYY_MM month code_of_country
DEFAULT 2023-06-18T18:37:08.481225Z 0 2020_07 7 GB
DEFAULT 2023-06-18T18:37:08.481230Z 1 2018_02 2 FI
DEFAULT 2023-06-18T18:37:08.481234Z 2 2022_02 2 CA
DEFAULT 2023-06-18T18:37:08.481238Z 3 2018_06 6 SE
DEFAULT 2023-06-18T18:37:08.481241Z 4 2020_01 1 DE
DEFAULT 2023-06-18T18:37:08.481245Z ... ... ... ...
DEFAULT 2023-06-18T18:37:08.481253Z 7766 2021_03 3 SE
DEFAULT 2023-06-18T18:37:08.481257Z 7767 2020_09 9 DE
DEFAULT 2023-06-18T18:37:08.481260Z 7768 2021_11 11 GB
DEFAULT 2023-06-18T18:37:08.481264Z 7769 2022_05 5 GB
DEFAULT 2023-06-18T18:37:08.481267Z 7770 2019_07 7 AU

我试过强迫

table_schema=[
    {"name": "SKU", "type": "STRING", "mode": "NULLABLE"},
    {"name": "text", "type": "STRING", "mode": "NULLABLE"},
    {"name": "document_date1", "type": "DATE", "mode": "NULLABLE"},
    {"name": "Sum_quantity1", "type": "FLOAT64", "mode": "NULLABLE"},
    {"name": "Landon", "type": "STRING", "mode": "NULLABLE"},
    {"name": "Category", "type": "STRING", "mode": "NULLABLE"},
    {"name": "Wheel", "type": "STRING", "mode": "NULLABLE"},
    {"name": "Frame", "type": "STRING", "mode": "NULLABLE"},
    {"name": " Version", "type": "STRING", "mode": "NULLABLE"},
    {"name": "Color", "type": "STRING", "mode": "NULLABLE"},
    {"name": "YYYY_MM", "type": "STRING", "mode": "NULLABLE"},
    {"name": "month", "type": "STRING", "mode": "NULLABLE"},
    {"name": "code_of_country ", "type": "STRING", "mode": "NULLABLE"},
],

但我也犯了同样的错误

xxls0lw8

xxls0lw81#

错误“pyarrow.lib.ArrowNotImplementedError:NumPyConverter doesn 't implement <list<item: int64>> conversion”表示数据框中有一列是整数列表,to_gbq函数不支持此转换。
要解决这个问题,您可以尝试在将列传递给to_gbq之前将其转换为字符串:

df['my_list_col'] = df['my_list_col'].astype(str)
pandas_gbq.to_gbq(
    df,
    destination,
    project_id,
    if_exists="replace"
)

或者,您可以尝试在将列传递给to_gbq之前将其转换为受支持的类型。例如,如果列表仅包含整数,则可以将其转换为逗号分隔的字符串:

df['my_list_col'] = df['my_list_col'].apply(lambda x: ','.join(x))
pandas_gbq.to_gbq(
    df,
    destination,
    project_id,
    if_exists="replace"
)

确保使用导致问题的列的名称更新my_list_col

相关问题