numpy 如何对包含列表的Pandas列进行one-hot编码?

jgzswidk  于 2022-12-23  发布在  其他
关注(0)|答案(6)|浏览(153)

我想把一个由元素列表组成的Pandas列分解成与唯一元素一样多的列,即one-hot-encode them(值1表示行中存在的给定元素,0表示不存在的情况)。
例如,以 Dataframe df为例

Col1   Col2         Col3
 C      33     [Apple, Orange, Banana]
 A      2.5    [Apple, Grape]
 B      42     [Banana]

我想将其转换为:

法国法郎

Col1   Col2   Apple   Orange   Banana   Grape
 C      33     1        1        1       0
 A      2.5    1        0        0       1
 B      42     0        0        1       0

我如何使用panda/sklearn来实现这一点?

wpx232ag

wpx232ag1#

我们也可以使用sklearn.preprocessing.MultiLabelBinarizer
我们经常希望对真实世界的数据使用sparseDataFrame,以节省大量RAM。

稀疏解(适用于Pandas v0.25.0+)

from sklearn.preprocessing import MultiLabelBinarizer

mlb = MultiLabelBinarizer(sparse_output=True)

df = df.join(
            pd.DataFrame.sparse.from_spmatrix(
                mlb.fit_transform(df.pop('Col3')),
                index=df.index,
                columns=mlb.classes_))

结果:

In [38]: df
Out[38]:
  Col1  Col2  Apple  Banana  Grape  Orange
0    C  33.0      1       1      0       1
1    A   2.5      1       0      1       0
2    B  42.0      0       1      0       0

In [39]: df.dtypes
Out[39]:
Col1                object
Col2               float64
Apple     Sparse[int32, 0]
Banana    Sparse[int32, 0]
Grape     Sparse[int32, 0]
Orange    Sparse[int32, 0]
dtype: object

In [40]: df.memory_usage()
Out[40]:
Index     128
Col1       24
Col2       24
Apple      16    #  <--- NOTE!
Banana     16    #  <--- NOTE!
Grape       8    #  <--- NOTE!
Orange      8    #  <--- NOTE!
dtype: int64

浓溶液

mlb = MultiLabelBinarizer()
df = df.join(pd.DataFrame(mlb.fit_transform(df.pop('Col3')),
                          columns=mlb.classes_,
                          index=df.index))

结果:

In [77]: df
Out[77]:
  Col1  Col2  Apple  Banana  Grape  Orange
0    C  33.0      1       1      0       1
1    A   2.5      1       0      1       0
2    B  42.0      0       1      0       0
w8ntj3qf

w8ntj3qf2#

    • 备选案文1**
  • 简短回答 *

第一个月

df.drop('Col3', 1).join(df.Col3.str.join('|').str.get_dummies())

  Col1  Col2  Apple  Banana  Grape  Orange
0    C  33.0      1       1      0       1
1    A   2.5      1       0      1       0
2    B  42.0      0       1      0       0
    • 备选案文2**
  • 快速回答 *

pir_fast

v = df.Col3.values
l = [len(x) for x in v.tolist()]
f, u = pd.factorize(np.concatenate(v))
n, m = len(v), u.size
i = np.arange(n).repeat(l)

dummies = pd.DataFrame(
    np.bincount(i * m + f, minlength=n * m).reshape(n, m),
    df.index, u
)

df.drop('Col3', 1).join(dummies)

  Col1  Col2  Apple  Orange  Banana  Grape
0    C  33.0      1       1       1      0
1    A   2.5      1       0       0      1
2    B  42.0      0       0       1      0
    • 备选案文3**

pir_alt1

df.drop('Col3', 1).join(
    pd.get_dummies(
        pd.DataFrame(df.Col3.tolist(), df.index).stack()
    ).astype(int).groupby(level=0).sum()
)

  Col1  Col2  Apple  Orange  Banana  Grape
0    C  33.0      1       1       1      0
1    A   2.5      1       0       0      1
2    B  42.0      0       0       1      0
    • 计时结果**
  • 以下代码 *

def maxu(df):
    mlb = MultiLabelBinarizer()
    d = pd.DataFrame(
        mlb.fit_transform(df.Col3.values)
        , df.index, mlb.classes_
    )
    return df.drop('Col3', 1).join(d)

def bos(df):
    return df.drop('Col3', 1).assign(**pd.get_dummies(df.Col3.apply(lambda x:pd.Series(x)).stack().reset_index(level=1,drop=True)).sum(level=0))

def psi(df):
    return pd.concat([
        df.drop("Col3", 1),
        df.Col3.apply(lambda x: pd.Series(1, x)).fillna(0)
    ], axis=1)

def alex(df):
    return df[['Col1', 'Col2']].assign(**{fruit: [1 if fruit in cell else 0 for cell in df.Col3] 
                                       for fruit in set(fruit for fruits in df.Col3 
                                                        for fruit in fruits)})

def pir_slow(df):
    return df.drop('Col3', 1).join(df.Col3.str.join('|').str.get_dummies())

def pir_alt1(df):
    return df.drop('Col3', 1).join(pd.get_dummies(pd.DataFrame(df.Col3.tolist()).stack()).astype(int).sum(level=0))

def pir_fast(df):
    v = df.Col3.values
    l = [len(x) for x in v.tolist()]
    f, u = pd.factorize(np.concatenate(v))
    n, m = len(v), u.size
    i = np.arange(n).repeat(l)

    dummies = pd.DataFrame(
        np.bincount(i * m + f, minlength=n * m).reshape(n, m),
        df.index, u
    )

    return df.drop('Col3', 1).join(dummies)

results = pd.DataFrame(
    index=(1, 3, 10, 30, 100, 300, 1000, 3000),
    columns='maxu bos psi alex pir_slow pir_fast pir_alt1'.split()
)

for i in results.index:
    d = pd.concat([df] * i, ignore_index=True)
    for j in results.columns:
        stmt = '{}(d)'.format(j)
        setp = 'from __main__ import d, {}'.format(j)
        results.set_value(i, j, timeit(stmt, setp, number=10))
3yhwsihp

3yhwsihp3#

您可以使用函数explode(0.25.0版中的新增函数)和crosstab

s = df['Col3'].explode()
df[['Col1', 'Col2']].join(pd.crosstab(s.index, s))

或者在Python 3.7+中:

df[['Col1', 'Col2']].join(pd.crosstab((s:=df['Col3'].explode()).index, s))

使用方法isin的另一种方法:

from itertools import chain

lst = sorted(set(chain.from_iterable(df['Col3'])))
s = pd.Series(lst, index=lst)
df.join(df.pop('Col3').apply(lambda x: s.isin(x)).astype(int))

输出:

Col1  Col2  Apple  Banana  Grape  Orange
0    C  33.0      1       1      0       1
1    A   2.5      1       0      1       0
2    B  42.0      0       1      0       0
qoefvg9y

qoefvg9y4#

使用get_dummies

df_out = df.assign(**pd.get_dummies(df.Col3.apply(lambda x:pd.Series(x)).stack().reset_index(level=1,drop=True)).sum(level=0))

输出:

Col1  Col2                     Col3  Apple  Banana  Grape  Orange
0    C  33.0  [Apple, Orange, Banana]      1       1      0       1
1    A   2.5           [Apple, Grape]      1       0      1       0
2    B  42.0                 [Banana]      0       1      0       0

清除色谱柱:

df_out.drop('Col3',axis=1)

输出:

Col1  Col2  Apple  Banana  Grape  Orange
0    C  33.0      1       1      0       1
1    A   2.5      1       0      1       0
2    B  42.0      0       1      0       0
yyyllmsg

yyyllmsg5#

您可以使用apply循环Col3,并将每个元素转换为Series,其中列表作为索引,成为结果数据框中的标题:

pd.concat([
        df.drop("Col3", 1),
        df.Col3.apply(lambda x: pd.Series(1, x)).fillna(0)
    ], axis=1)

#Col1   Col2    Apple   Banana  Grape   Orange
#0  C   33.0      1.0      1.0    0.0     1.0
#1  A    2.5      1.0      0.0    1.0     0.0
#2  B   42.0      0.0      1.0    0.0     0.0
bqjvbblv

bqjvbblv6#

可以使用集合解析得到Col3中的所有唯一结果,如下所示:

set(fruit for fruits in df.Col3 for fruit in fruits)

使用字典理解,你可以检查每一个独特的水果,看看它是否在列中。

>>> df[['Col1', 'Col2']].assign(**{fruit: [1 if fruit in cell else 0 for cell in df.Col3] 
                                   for fruit in set(fruit for fruits in df.Col3 
                                                    for fruit in fruits)})
  Col1  Col2  Apple  Banana  Grape  Orange
0    C  33.0      1       1      0       1
1    A   2.5      1       0      1       0
2    B  42.0      0       1      0       0

计时

dfs = pd.concat([df] * 1000)  # Use 3,000 rows in the dataframe.

# Solution 1 by @Alexander (me)
%%timeit -n 1000 
dfs[['Col1', 'Col2']].assign(**{fruit: [1 if fruit in cell else 0 for cell in dfs.Col3] 
                                for fruit in set(fruit for fruits in dfs.Col3 for fruit in fruits)})
# 10 loops, best of 3: 4.57 ms per loop

# Solution 2 by @Psidom
%%timeit -n 1000
pd.concat([
        dfs.drop("Col3", 1),
        dfs.Col3.apply(lambda x: pd.Series(1, x)).fillna(0)
    ], axis=1)
# 10 loops, best of 3: 748 ms per loop

# Solution 3 by @MaxU
from sklearn.preprocessing import MultiLabelBinarizer
mlb = MultiLabelBinarizer()

%%timeit -n 10 
dfs.join(pd.DataFrame(mlb.fit_transform(dfs.Col3),
                          columns=mlb.classes_,
                          index=dfs.index))
# 10 loops, best of 3: 283 ms per loop

# Solution 4 by @ScottBoston
%%timeit -n 10
df_out = dfs.assign(**pd.get_dummies(dfs.Col3.apply(lambda x:pd.Series(x)).stack().reset_index(level=1,drop=True)).sum(level=0))
# 10 loops, best of 3: 512 ms per loop

But...
>>> print(df_out.head())
  Col1  Col2                     Col3  Apple  Banana  Grape  Orange
0    C  33.0  [Apple, Orange, Banana]   1000    1000      0    1000
1    A   2.5           [Apple, Grape]   1000       0   1000       0
2    B  42.0                 [Banana]      0    1000      0       0
0    C  33.0  [Apple, Orange, Banana]   1000    1000      0    1000
1    A   2.5           [Apple, Grape]   1000       0   1000       0

相关问题