可以将结构数组构造为：

df2 = df.selectExpr(
    'id', 
    """
    array(
        struct('en' as locale, name_en as value),
        struct('sp' as locale, name_sp as value),
        struct('fr' as locale, name_fr as value)
    ) as names
    """
)

df2.show(truncate=False)
+---+------------------------------------------------------+
|id |names                                                 |
+---+------------------------------------------------------+
|1  |[[en, hello], [sp, hello_spanish], [fr, hello_french]]|
+---+------------------------------------------------------+

要获取json，可以尝试：

df2.toJSON().collect()

# this gives:

# ['{"id":1,"names":[{"locale":"en","value":"hello"},

# {"locale":"sp","value":"hello_spanish"},

# {"locale":"fr","value":"hello_french"}]}']