一种可能的方法是使用 mapParitions ```
import os
import numpy as np

swap = lambda x: (x[1], x[0])

def add_random_key(it):
# make sure we get a proper random seed
seed = int(os.urandom(4).encode('hex'), 16)
# create separate generator
rs = np.random.RandomState(seed)
# Could be randint if you prefer integers
return ((rs.rand(), swap(x)) for x in it)

rdd_with_keys = (rdd

It will be used as final key. If you don't accept gaps

use zipWithIndex but this should be cheaper

.zipWithUniqueId()
.mapPartitions(add_random_key, preservesPartitioning=True))

接下来可以重新分区、对每个分区排序并提取值：

n = rdd.getNumPartitions()
(rdd_with_keys
# partition by random key to put data on random partition
.partitionBy(n)
# Sort partition by random value to ensure random order on partition
.mapPartitions(sorted, preservesPartitioning=True)
# Extract (unique_id, value) pairs
.values())

如果每个分区的排序仍然很慢，可以用fisher-yates shuffle来代替。
如果您只是需要一个随机数据，那么您可以使用 `mllib.RandomRDDs` ```
from pyspark.mllib.random import RandomRDDs
RandomRDDs.uniformRDD(sc, n)

理论上它可以通过输入压缩 rdd 但它需要匹配每个分区的元素数。

Pypark成功了！

from random import randrange
data_rnd = data.sortBy(lambda x: randrange(1000000))