pandas pyarrow.lib.ArrowInvalid:('无法将X转换为类型Y:推断Arrow数据类型时无法识别Python值类型“)

uyhoqukh  于 2023-02-17  发布在  Python
关注(0)|答案(4)|浏览(696)

使用pyarrow将包含Player对象的pandas.DataFrame转换为pyarrow.Table,代码如下

import pandas as pd
import pyarrow as pa

class Player:
    def __init__(self, name, age, gender):
        self.name = name
        self.age = age
        self.gender = gender

    def __repr__(self):
        return f'<{self.name} ({self.age})>'

data = [
    Player('Jack', 21, 'm'),
    Player('Ryan', 18, 'm'),
    Player('Jane', 35, 'f'),
]
df = pd.DataFrame(data, columns=['player'])
print(pa.Table.from_pandas(df))

我们得到错误:

pyarrow.lib.ArrowInvalid: ('Could not convert <Jack (21)> with type Player: did not recognize Python value type when inferring an Arrow data type', 'Conversion failed for column 0 with type object')

使用时遇到相同的错误

df.to_parquet('players.pq')

pyarrow是否可以退而使用pickle序列化这些Python对象?或者有更好的解决方案吗?pyarrow.Table最终将使用Parquet.write_table()写入磁盘。

    • 使用Python 3.8.0,Pandas0.25.3,pyarrow 0.13.0.*
    • pandas.DataFrame.to_parquet()不支持多索引,因此首选使用pq.write_table(pa.Table.from_dataframe(pandas.DataFrame))的解决方案。*

谢谢大家!

kcrjzv8t

kcrjzv8t1#

我的建议是将数据插入到已经序列化的DataFrame中。

最佳选项-使用数据类(python〉=3.7)

装饰器将Player类定义为数据类,并让序列化在本地完成(到JSON)。

import pandas as pd
from dataclasses import dataclass

@dataclass
class PlayerV2:
    name:str
    age:int
    gender:str

    def __repr__(self):
        return f'<{self.name} ({self.age})>'

dataV2 = [
    PlayerV2(name='Jack', age=21, gender='m'),
    PlayerV2(name='Ryan', age=18, gender='m'),
    PlayerV2(name='Jane', age=35, gender='f'),
]

# The serialization is done natively to JSON
df_v2 = pd.DataFrame(data, columns=['player'])
print(df_v2)

# Can still get the objects's attributes by deserializeing the record
json.loads(df_v2["player"][0])['name']

手动序列化对象(python〈3.7)

在Player类中定义一个序列化函数,并在创建Dataframe之前序列化每个示例。

import pandas as pd
import json

class Player:
    def __init__(self, name, age, gender):
        self.name = name
        self.age = age
        self.gender = gender

    def __repr__(self):
        return f'<{self.name} ({self.age})>'
    
    # The serialization function for JSON, if for some reason you really need pickle you can use it instead
    def toJSON(self):
        return json.dumps(self, default=lambda o: o.__dict__)

# Serialize the objects before inserting it into the DataFrame
data = [
    Player('Jack', 21, 'm').toJSON(),
    Player('Ryan', 18, 'm').toJSON(),
    Player('Jane', 35, 'f').toJSON(),
]
df = pd.DataFrame(data, columns=['player'])

# You can see all the data inserted as a serialized json into the column player
print(df)

# Can still get the objects's attributes by deserializeing the record
json.loads(df["player"][0])['name']
hrysbysz

hrysbysz2#

在我的理解中,由于repr,“type”存在问题。尝试此方法(有效):

class Player:
    def __init__(self, name, age, gender):
        self.name = name
        self.age = age
        self.gender = gender

    def other(self):
        return f'<{self.name} ({self.age})>'

data = [
    Player('Jack', 21, 'm').other(),
    Player('Ryan', 18, 'm').other(),
    Player('Jane', 35, 'f').other(),
]
df = pd.DataFrame(data, columns=['player'])
print(df)
        player
0  <Jack (21)>
1  <Ryan (18)>
2  <Jane (35)>

print(pa.Table.from_pandas(df))

pyarrow.Table
player: string
wfsdck30

wfsdck303#

不确定Parquet是否支持〈string(int)〉格式,但它可以在dict和list上工作。
对于python类。通过调用object.dict来获取对象的字典表示。
例如以下作品

from dataclasses import dataclass
import pandas as pd
import pyarrow as pa

@dataclass
class Player:
  name: str
  age: int
  gender: str

players = [
  {"name": "player1", "age": 12, "gender": "f"},
  {"name": "player2", "age": 22, "gender": "m"},
  {"name": "player3", "age": 18, "gender": "m"}
]
df = pd.DataFrame()
df["players"] = [Player(**r).__dict__ for r in players]

pa.Table.from_pandas(df)
xdnvmnnf

xdnvmnnf4#

另一种选择是用你自己的自定义Dtype扩展Pandas。Pandas给出了相当多关于如何创建扩展Dtype的文档,你可以查看base class了解更多细节,以及existing extensions的例子。
也就是说,这有点复杂,如果您希望解决“could not convert”错误,并将数据打印或保存到parquet,我建议您使用其他答案中提到的某种形式的预序列化,或者在类上实现__str__,然后将列类型转换为str。由于__str__将用于其预期用途,因此可以改进__repr__,使其返回一个类似于有效Python表达式的字符串,该表达式可用于重新创建具有相同值的对象(给定适当的环境)。

import pandas as pd
import pyarrow as pa

class Player:
    def __init__(self, name, age, gender):
        self.name = name
        self.age = age
        self.gender = gender

    def __repr__(self):
        return f'Player("{self.name}", {self.age}, "{self.gender}")'

    def __str__(self):
        return f'<{self.name} ({self.age})>'

data = [
    Player('Jack', 21, 'm'),
    Player('Ryan', 18, 'm'),
    Player('Jane', 35, 'f'),
]
df = pd.DataFrame(data, columns=['player'])
for col in [c for c in df.select_dtypes(include=['object']).columns]:
    df[col] = df[col].astype('str')

print(pa.Table.from_pandas(df))
df.to_parquet('players.pq')
print([repr(d) for d in data])

这给出了输出:

pyarrow.Table
player: string
----
player: [["<Jack (21)>","<Ryan (18)>","<Jane (35)>"]]
# No output from to_parquet b/c there was no error
['Player("Jack", 21, "m")', 'Player("Ryan", 18, "m")', 'Player("Jane", 35, "f")']

当然,如果您希望保留原始的DataFrame和原始的类型,您将希望在副本上而不是在原始文件上更改这些列类型。

相关问题