pandas—在python中将json数据加载到Dataframe中

mm9b1k5b 于 2021-07-14 发布在 Spark

关注(0)|答案(4)|浏览(395)

list = {'masterId': 2, 'name': 'name', 'description': 'xyz', 'signalTypeRefId': 4, 'unitOfMeasureRefId': 1, 'precision': 1, 'min': -125, 'max': 125, 'isDeprecated': False}

我需要将上面的json加载到一个dataframe中，并尝试了以下操作，但没有成功

df = pd.DataFrame.from_dict(list, orient = 'index')
display(df)

错误：
typeerror:字段0:无法合并类型<class'pyspark.sql.types.longtype'>和<class'pyspark.sql.types.stringtype'>

python apache-spark pyspark pandas

来源：https://stackoverflow.com/questions/67098648/load-json-data-into-a-dataframe-in-python

4条答案

按热度按时间

plicqrtu1#

在创建Dataframe之前，需要将字典 Package 到列表中：

data = {'masterId': 2, 'name': 'name', 'description': 'xyz', 'signalTypeRefId': 4, 'unitOfMeasureRefId': 1, 'precision': 1, 'min': -125, 'max': 125, 'isDeprecated': False}

df = spark.createDataFrame([data])

df.show()
+-----------+------------+--------+---+----+----+---------+---------------+------------------+
|description|isDeprecated|masterId|max| min|name|precision|signalTypeRefId|unitOfMeasureRefId|
+-----------+------------+--------+---+----+----+---------+---------------+------------------+
|        xyz|       false|       2|125|-125|name|        1|              4|                 1|
+-----------+------------+--------+---+----+----+---------+---------------+------------------+

或者，您可以将其转换为pandasDataframe，并从中创建sparkDataframe，但仍需要将字典 Package 到列表中：

data = {'masterId': 2, 'name': 'name', 'description': 'xyz', 'signalTypeRefId': 4, 'unitOfMeasureRefId': 1, 'precision': 1, 'min': -125, 'max': 125, 'isDeprecated': False}

df = spark.createDataFrame(pd.DataFrame([data]))

df.show()
+--------+----+-----------+---------------+------------------+---------+----+---+------------+
|masterId|name|description|signalTypeRefId|unitOfMeasureRefId|precision| min|max|isDeprecated|
+--------+----+-----------+---------------+------------------+---------+----+---+------------+
|       2|name|        xyz|              4|                 1|        1|-125|125|       false|
+--------+----+-----------+---------------+------------------+---------+----+---+------------+

赞(0）回复(0）举报 2021-07-14

oxalkeyp2#

data = pd.DataFrame([list])

有关将json转换为Dataframe的更多信息，请查看下面的链接：）
https://pandas.pydata.org/docs/reference/api/pandas.read_json.html

赞(0）回复(0）举报 2021-07-14

knsnq2tg3#

dct = {'masterId': 2, 'name': 'name', 'description': 'xyz', 'signalTypeRefId': 4, 'unitOfMeasureRefId': 1, 'precision': 1, 'min': -125, 'max': 125, 'isDeprecated': False}

df = pd.DataFrame.from_dict(dct, orient="index")
display(df)

"""
                        0
masterId                2
name                 name
description           xyz
signalTypeRefId         4
unitOfMeasureRefId      1
precision               1
min                  -125
max                   125
isDeprecated        False
"""

要将其作为一行，请使用 .transpose() ```
df.transpose()
"""
Out[15]:
masterId name description ... min max isDeprecated
0 2 name xyz ... -125 125 False
"""

赞(0）回复(0）举报 2021-07-14

zqdjd7g94#