对DataFrame进行切片的成本很高。使用python和sets对我来说快了5倍：

num_neg_samples = 4
targets = set(pair_sample['TARGET_ID'])
negids = [rng.choice(list(targets-{t}), size=num_neg_samples, replace=False)
          for t in pair_sample['TARGET_ID']]

输出量：

[array([563401, 169859, 180204, 953531]),
 array([310634,  27685, 992711, 801847]),
 array([519938, 717685, 150933, 466746]),
 array([496852, 370208, 930286, 293928]),
 ...
]

使用python的random.sample快10倍：

num_neg_samples = 4
targets = set(pair_sample['TARGET_ID'])
negids = [random.sample(list(targets-{t}), k=num_neg_samples)
          for t in pair_sample['TARGET_ID']]

您可以通过使用concurrent.futures.ThreadPoolExecutor在并行线程中收集过滤后的负样本来加快处理速度。我在我的机器上获得了4-5倍的速度提升（初始方法：~93秒，螺纹：约20秒）：

import time
from concurrent import futures
from functools import partial
def select_neg_sample(target_id, all_pairs, size=4):
    return rng.choice(
        all_pairs.loc[all_pairs['TARGET_ID'] != target_id, 'TARGET_ID'].values,
        size=size
    )
start = time.time()
targets = np.repeat(pair_sample['TARGET_ID'].values, (num_neg_samples + 1), axis=0)
contexts = pair_sample['CONTEXT_ID']
print(f'{len(pair_sample["TARGET_ID"])} target_ids to create neg samples for')
with futures.ThreadPoolExecutor() as executor:
    it = executor.map(partial(select_neg_sample, all_pairs=all_pairs),
                      pair_sample['TARGET_ID'])
    neg_ids = list(it)
print(time.time() - start, 'seconds')
print(f'neg_ids size: {len(neg_ids)}, last entry: {neg_ids[-1]}')

TARGET_ID  CONTEXT_ID
294231     704025      731665
22308      430988      862487
484542      70459      196505
119601     762881      339710
309284     459537      433841
10240 target_ids to create neg samples for
19.128324270248413 seconds
neg_ids size: 10240, last entry: [ 33955 538037 312413 692326]

根据@mozway的回答和这个，我在一个相关的帖子中发现：https://stackoverflow.com/a/44349133/5240684，我想出了以下，这是足够的我的情况：

import pandas as pd
import numpy as np
import time
num_neg_samples = 4
# sample data
rng = np.random.default_rng()
all_pairs = pd.DataFrame({
    'TARGET_ID': rng.choice(100_000, size=500_000, replace=True),
    'CONTEXT_ID': rng.choice(100_000, size=500_000, replace=True)
})
pair_sample = all_pairs.sample(10240)
print(pair_sample.head())
start = time.time()
unique_targets = all_pairs['TARGET_ID'].unique()
# index to be able to use range
unique_targets_indexed = dict(zip(unique_targets, range(len(unique_targets))))
        
targets = pair_sample['TARGET_ID'].values
contexts = pair_sample['CONTEXT_ID'].values
negative_targets = np.repeat(pair_sample['TARGET_ID'].values,(num_neg_samples),axis=0)
negative_contexts = [unique_targets[random_neg(0,len(unique_targets), unique_targets_indexed[t])] for t in negative_targets]
print(f'\nTook {time.time() - start} seconds')

测试结果：

TARGET_ID  CONTEXT_ID
284483      70099       74132
164208       3090       13895
186524      89871       81281
270224      46461       56410
470391       7434       81272
Took 0.19039297103881836 seconds

即使从1 mio不同的目标id中采样，我的机器上的处理时间也增加到只有大约0.31秒。
基本思想是将目标id的索引设置为从0到len（unique_targets）的范围，这样我们就可以从一个范围而不是从一个集合中采样一个随机整数，因为这样更快。
这样，我最终得到batch = (contexts, targets, negative_contexts, negative_targets)作为我的批处理。
谢谢你给我指明了正确的方向。