python-3.x Pandas json_normalize的逆运算

6ovsh4lw  于 2022-11-19  发布在  Python
关注(0)|答案(6)|浏览(165)

我刚刚发现了json_normalize函数,它可以很好地处理JSON对象并生成一个pandas Dataframe。现在我需要反向操作,它可以处理相同的Dataframe并生成一个json(或者类似于json的字典,我可以很容易地将其转换为json),其结构与原始json相同。
下面是一个例子:https://hackersandslackers.com/json-into-pandas-dataframes/
他们接受一个JSON对象(或类似JSON的python字典)并将其转换为一个 Dataframe ,但现在我想接受该 Dataframe 并将其转换回一个类似JSON的字典(以便稍后转储到json文件)。

bzzcjhmw

bzzcjhmw1#

我用几个函数实现了它

def set_for_keys(my_dict, key_arr, val):
    """
    Set val at path in my_dict defined by the string (or serializable object) array key_arr
    """
    current = my_dict
    for i in range(len(key_arr)):
        key = key_arr[i]
        if key not in current:
            if i==len(key_arr)-1:
                current[key] = val
            else:
                current[key] = {}
        else:
            if type(current[key]) is not dict:
                print("Given dictionary is not compatible with key structure requested")
                raise ValueError("Dictionary key already occupied")

        current = current[key]

    return my_dict

def to_formatted_json(df, sep="."):
    result = []
    for _, row in df.iterrows():
        parsed_row = {}
        for idx, val in row.iteritems():
            keys = idx.split(sep)
            parsed_row = set_for_keys(parsed_row, keys, val)

        result.append(parsed_row)
    return result

#Where df was parsed from json-dict using json_normalize
to_formatted_json(df, sep=".")
ukdjmx9f

ukdjmx9f2#

更简单的方法:

  • 仅使用1个函数...*
def df_to_formatted_json(df, sep="."):
    """
    The opposite of json_normalize
    """
    result = []
    for idx, row in df.iterrows():
        parsed_row = {}
        for col_label,v in row.items():
            keys = col_label.split(".")

            current = parsed_row
            for i, k in enumerate(keys):
                if i==len(keys)-1:
                    current[k] = v
                else:
                    if k not in current.keys():
                        current[k] = {}
                    current = current[k]
        # save
        result.append(parsed_row)
    return result
nfeuvbwi

nfeuvbwi3#

df.to_json(path)

df.to_dict()
vcudknz3

vcudknz34#

让我把我的意见说出来
在向后转换之后,你可能需要从你生成的jsons中删除空列。因此,我检查了瓦尔!= np. nan。但是你不能直接这样做,相反,你需要检查val == val与否,因为np.nan!=本身。我的版本:

def to_formatted_json(df, sep="."):
    result = []
    for _, row in df.iterrows():
        parsed_row = {}
        for idx, val in row.iteritems():
            if val == val:
                keys = idx.split(sep)
                parsed_row = set_for_keys(parsed_row, keys, val)

        result.append(parsed_row)
    return result
b09cbbtk

b09cbbtk5#

这是一个在我看来很有效的解决方案。它被设计成在只有一行的 Dataframe 上工作,但是它可以很容易地在大的 Dataframe 上循环。

class JsonRecreate():
    
    def __init__(self, df):
        self.df = df

    def pandas_to_json(self):
        df = self.df
        # determine the number of nesting levels
        number_levels = np.max([len(i.split('.')) for i in df.columns])
        # put all the nesting levels in an a list
        levels = []
        for level_idx in np.arange(number_levels):
            levels.append(np.array([i.split('.')[level_idx] if len(i.split('.')) > level_idx else ''
                                    for i in df.columns.tolist()]))
        self.levels = levels
        return self.create_dict(upper_bound = self.levels[0].shape[0])

    def create_dict(self, level_idx = 0, lower_bound = 0, upper_bound = 100):
        ''' Function to create the dictionary starting from a pandas dataframe generated by json_normalize '''
        levels = self.levels
        dict_ = {}
        # current nesting level
        level = levels[level_idx]
        # loop over all the relevant elements of the level (relevant w.r.t. its parent)
        for key in [i for i in np.unique(level[lower_bound: upper_bound]) if i != '']:
            # find where a particular key occurs in the level
            correspondence = np.where(level[lower_bound: upper_bound] == key)[0] + lower_bound
            # check if the value(s) corresponding to the key appears once (multiple times)
            if correspondence.shape[0] == 1:
                # if the occurence is unique, append the value to the dictionary
                dict_[key] = self.df.values[0][correspondence[0]]
            else:
                # otherwhise, redefine the relevant bounds and call the function recursively
                lower_bound_, upper_bound_ = correspondence.min(), correspondence.max() + 1
                dict_[key] = self.create_dict(level_idx + 1, lower_bound_, upper_bound_)
        return dict_

我用一个简单的 Dataframe 进行了测试,例如:

df = pd.DataFrame({'a.b': [1], 'a.c.d': [2], 'a.c.e': [3], 'a.z.h1': [-1], 'a.z.h2': [-2], 'f': [4], 'g.h': [5], 'g.i.l': [6], 'g.i.m': [7], 'g.z.h1': [-3], 'g.z.h2': [-4]})

json中的顺序并不完全保留在结果json中,但如果需要,可以很容易地处理。

oyjwcjzk

oyjwcjzk6#

我只是用两个函数实现了这个。
1.从DataFrame中获取属于嵌套字段一部分的字段的完整列表。仅父级,即如果location.city.code符合条件,则只关注location.city。按嵌套的最深级别对其进行排序,即location.citylocation嵌套得更深。
1.从嵌套最深的父字段开始,通过在列名中搜索来查找所有子字段。在DataFrame中为父字段创建一个字段,该字段是所有子字段的组合(重命名以使其失去嵌套结构,例如location.city.code变为code),这些子字段转换为JSON,然后加载为字典值。最后,删除所有子字段。

def _get_nested_fields(df: pd.DataFrame) -> List[str]:
    """Return a list of nested fields, sorted by the deepest level of nesting first."""
    nested_fields = [*{field.rsplit(".", 1)[0] for field in df.columns if "." in field}]
    nested_fields.sort(key=lambda record: len(record.split(".")), reverse=True)
    return nested_fields

def df_denormalize(df: pd.DataFrame) -> pd.DataFrame:
    """
    Convert a normalised DataFrame into a nested structure.

    Fields separated by '.' are considered part of a nested structure.
    """
    nested_fields = _get_nested_fields(df)
    for field in nested_fields:
        list_of_children = [column for column in df.columns if field in column]
        rename = {
            field_name: field_name.rsplit(".", 1)[1] for field_name in list_of_children
        }
        renamed_fields = df[list_of_children].rename(columns=rename)
        df[field] = json.loads(renamed_fields.to_json(orient="records"))
        df.drop(list_of_children, axis=1, inplace=True)
    return df

相关问题