python 我的多处理线程池比单线程实现需要更长的时间来完成任务

sqxo8psd  于 2023-01-19  发布在  Python
关注(0)|答案(1)|浏览(147)

我写了一个算法,并试图比较不同版本的性能。我的基准函数使用线程池,但它需要相同的时间或比单核实现更长。我用pypy和python,版本3.11和结果是相同的。
要进行基准测试的方法:

def main(print_results=True):
    results = Queue()
    start_time = time.time()

    words = get_set_from_dict_file("usa.txt")
    results.put(f"Total words read: {len(words)}")
    results.put(f"Total time taken to read the file: {round((time.time() - start_time) * 1000)} ms")
    start_time_2 = time.time()

    pairs = getPairs(words)
    results.put(f"Number of words that can be built with 3 letter word + letter + 3 letter word: {len(pairs)}")

    results.put(f"Total time taken to find the pairs: {round((time.time() - start_time_2) * 1000)} ms")

    results.put(f"Time taken: {round((time.time() - start_time) * 1000)}ms")

    if print_results:
        [print(x) for x in results.queue]
    return (time.time() - start_time) * 1000

多线程线程池:

def benchmark(n=1000):
    # start number of threads equal to 90% of cores running main() using multiprocessing, continue until n runs complete
    core_count = os.cpu_count()
    thread_num = floor(core_count * 0.9)
    pool = ThreadPool(thread_num)

    results = pool.map_async(main, [False] * n)
    results = results.get()
    pool.close()
    avg_time_ms = round(sum(results) / len(results))
    # Save best run time and its code as a pickle file in format (time, code)
    # Currently hidden code
    return avg_time_ms, -1

试验:

if __name__ == "__main__":
    print("Do you want to benchmark? (y/n)")
    if input().upper() == "Y":
        print("Benchmark n times: (int)")
        n = input()
        n = int(n) if (n.isdigit() and 0 < int(n) <= 1000) else 100
        start = time.time()
        bench = benchmark(n)
        end = time.time()
        print("\n----------Multi-Thread Benchmark----------")
        print(f"Average time taken: {bench[0]} ms")
        print(f"Best time taken yet: {bench[1]} ms")
        print(f"Total bench time: {end - start:0.5} s")

        start = time.time()
        non_t_results = [main(False) for _ in range(n)]
        end = time.time()
        print("\n----------Single-Thread Benchmark----------")
        print(f"Average time taken: {round(sum(non_t_results) / len(non_t_results))} ms")
        print(f"Total bench time: {end - start:0.5} s")

    else:
        main()

每次运行它时,无论池中运行的线程数或线程数如何,池的完成速度都不会更快。

Do you want to benchmark? (y/n)
y
Benchmark n times: (int)
50

----------Multi-Thread Benchmark----------
Average time taken: 276 ms
Best time taken yet: -1 ms
Total bench time: 2.2814 s

----------Single-Thread Benchmark----------
Average time taken: 36 ms
Total bench time: 1.91 s

Process finished with exit code 0

我希望线程池完成得更快。

mbskvtky

mbskvtky1#

原来我用的是线程而不是进程。多亏了评论者,我才能理解ThreadPool是用于并发处理的,Pool是用于并行处理的。
以下是更改后的基准:

def benchmark(n=1000):
    # start number of threads equal to 90% of cores running main() using multiprocessing, continue until n runs complete
    core_count = os.cpu_count()
    process_num = floor(core_count * 0.9)

    with Pool(process_num) as pool:
        results = pool.map_async(main, [False] * n)
        results = results.get()
    avg_time_ms = round(sum(results) / len(results))
    # Save best run time and its code as a pickle file in format (time, code)
    """..."""
    return avg_time_ms, -1

相关问题