python 从panda Dataframe 创建BigQuery表,无需显式指定模式

ztyzrc3y  于 2022-12-17  发布在  Python
关注(0)|答案(3)|浏览(105)

我有一个panda Dataframe ,想从它创建一个BigQuery表。我知道有很多帖子都在问这个问题,但到目前为止,我能找到的所有答案都需要显式地指定每一列的模式。例如:

from google.cloud import bigquery as bq

client = bq.Client()

dataset_ref = client.dataset('my_dataset', project = 'my_project')
table_ref = dataset_ref.table('my_table')  

job_config = bq.LoadJobConfig( 
 schema=[ 
     bq.SchemaField("a", bq.enums.SqlTypeNames.STRING),
     bq.SchemaField("b", bq.enums.SqlTypeNames.INT64), 
     bq.SchemaField("c", bq.enums.SqlTypeNames.FLOAT64),         
 ]
) 

client.load_table_from_dataframe(my_df, table_ref, job_config=job_config).result()

然而,有时候我有一个包含很多列的 Dataframe (例如,100列),指定所有的列真的很不容易。有没有一种方法可以有效地做到这一点?
顺便说一句,我发现这个职位与类似的问题:Efficiently write a Pandas dataframe to Google BigQuery但似乎bq.Schema.from_dataframe不存在:

AttributeError: module 'google.cloud.bigquery' has no attribute 'Schema'
mspsb9vt

mspsb9vt1#

下面是将DataFrame加载到BQ的代码片段:

import pandas as pd
from google.cloud import bigquery

# Example data
df = pd.DataFrame({'a': [1,2,4], 'b': ['123', '456', '000']})

# Load client
client = bigquery.Client(project='your-project-id')

# Define table name, in format dataset.table_name
table = 'your-dataset.your-table'

# Load data to BQ
job = client.load_table_from_dataframe(df, table)

如果只想指定架构的子集,但仍导入所有列,则可以使用切换最后一行

# Define a job config object, with a subset of the schema
job_config = bigquery.LoadJobConfig(schema=[bigquery.SchemaField('b', 'STRING')])

# Load data to BQ
job = client.load_table_from_dataframe(df, table, job_config=job_config)
8hhllhi2

8hhllhi22#

下面是工作代码:

from google.cloud import bigquery
import pandas as pd

bigqueryClient = bigquery.Client()
tableRef = bigqueryClient.dataset("dataset-name").table("table-name")

dataFrame = pd.read_csv("file-name")

bigqueryJob = bigqueryClient.load_table_from_dataframe(dataFrame, tableRef)
bigqueryJob.result()
mrwjdhj3

mrwjdhj33#

现在只需安装pandas-gbq==0.18.1,然后

df.to_gbq(
        destination_table="my_project_id.my_dataset.my_table",
        project_id="my_project_id",
        credentials=service_account.Credentials.from_service_account_info(
            my_service_account_info     # there are several ways to authenticate
        ),
    )

文件:https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_gbq.html
有关身份验证说明,请参见How to authenticate with Google BigQuery guide

相关问题