postgresql 在Postgres表中插入字典列表,其中每个字典都有不同的键

cbjzeqam  于 2023-06-05  发布在  PostgreSQL
关注(0)|答案(3)|浏览(359)

我有一个Python字典列表,看起来像下面这样,每个字典可以有稍微不同的键。

data = [
  {'name': 'Bob', 'age': 32},
  {'name': 'Sara', 'city': 'Dallas'},
  {'name': 'John', 'age': 45, 'city': 'Atlanta'}
]

我还有一个Postgres表,它包含了在这个字典列表中看到的所有可能的键(例如:nameagecity)。
我正在寻找一个优雅的解决方案,以有效地将这些数据插入到我的数据库。虽然我可以逐行迭代data,并单独插入每一行,但这并不能很好地扩展到包含数百万条记录的实际数据集。
我尝试使用psycopg2中的execute_values函数,如下面的示例所示,但这要求所有字典都具有相同的键。

如何编辑下面的过程,以便一次插入多个字典,其中每个字典可以包含不同的键?

import psycopg2
from psycopg2.extras import execute_values

# connect to the database
conn = psycopg2.connect(
    host="localhost",
    database="db_name",
    user="psql_user",
    password="psql_password",
)
conn.autocommit = True
cur = conn.cursor()

# get the columns from first dictionary
columns = data[0].keys()

# write the SQL query to insert the records
query = """INSERT INTO schema.table 
            ({}) VALUES %s
            ON CONFLICT (name) DO NOTHING""".format(
        ",".join(columns)
)

# extract the values from each dictionary into as list of lists
values = [[value for value in line.values()] for line in data]

# execute the SQL query with the associated values
execute_values(cur, query, values)
zqdjd7g9

zqdjd7g91#

使用psycopg2sql模块。

create table books (id serial, bookcode integer, bookname text);

import psycopg2
from psycopg2 import sql

data = [{'bookcode': 1, 'bookname': 'test'}, {'bookcode': 2}]

con = psycopg2.connect("dbname=test host=localhost  user=postgres")
cur = con.cursor()

for d in data:
    col_names = list(d.keys())
    print(col_names)
    insert_qry = sql.SQL("insert into books ({})  values({})").format(sql.SQL(",").join(map(sql.Identifier, col_names)), 
                         sql.SQL(",").join(map(sql.Placeholder, col_names)))
    cur.execute(insert_qry, d)
con.commit()

select * from books;
 id | bookcode | bookname 
----+----------+----------
  1 |        1 | test
  2 |        2 | NULL

这当然假设列可以接受NULL值。如果不是这种情况,那么您需要为缺少的字段创建一些适当的值。

bkkx9g8r

bkkx9g8r2#

用一个模式合并字典(所有列都包含None的字典):

#...

data = [
  {'name': 'Bob', 'age': 32},
  {'name': 'Sara', 'city': 'Dallas'},
  {'name': 'John', 'age': 45, 'city': 'Atlanta'}
]

# pattern - all columns with None as value
nulls = {'name': None, 'age': None, "city": None}

# get the columns from the pattern
columns = nulls.keys()

# write the SQL query to insert the records
query = """INSERT INTO schema.table
            ({}) VALUES %s
            ON CONFLICT (name) DO NOTHING""".format(
        ",".join(columns)
)

# extract the values from each dictionary into as list of lists
values = [[value for value in (nulls | line).values()] for line in data]

# execute the SQL query with the associated values
execute_values(cur, query, values)
eanckbw9

eanckbw93#

假设你知道完整的列列表(如果你正在填写一个表,你应该知道),你可以转换你的数据来填写缺失的值:

data = [
  {'name': 'Bob', 'age': 32},
  {'name': 'Sara', 'city': 'Dallas'},
  {'name': 'John', 'age': 45, 'city': 'Atlanta'}
]

import psycopg2
from psycopg2.extras import execute_values

# connect to the database
conn = psycopg2.connect(
    host="localhost",
    database="db_name",
    user="psql_user",
    password="psql_password",
)
conn.autocommit = True
cur = conn.cursor()

columns = ['name','age','city','state','zip']
empty = {k:None for k in columns}

# get the columns from first dictionary

# write the SQL query to insert the records
query = """INSERT INTO schema.table 
            ({}) VALUES ({})
            ON CONFLICT (name) DO NOTHING""".format(
        ",".join(columns),
        ','.join('%' for _ in columns)
)

# extract the values from each dictionary into as list of lists

values = []
for row in data:
    x = empty.copy()
    x.update( row )
    values.append( x )
print(query)
print(values)

# execute the SQL query with the associated values
#execute_values(cur, query, values)

相关问题