在shell中一次并行运行不超过4个进程

x33g5p2x  于 2022-11-16  发布在  Shell
关注(0)|答案(3)|浏览(206)

目前,我一直在高效地运行耗时的模拟。我的目的是并行运行4个模拟,因为这是一个单线程应用程序和一个四核系统。我不得不使用shell脚本的变体:

./sim -r 1 &
./sim -r 2 &
./sim -r 3 &
./sim -r 4 &
wait
./sim -r 5 &
./sim -r 6 &
./sim -r 7 &
./sim -r 8 &
wait
... (another 112 jobs)

使用这段代码时,会反复等待。我还尝试将任务分成四个脚本并运行每个脚本,结果是一个脚本完成,而另一个脚本有大约30%的剩余作业。我无法预测模拟需要多长时间。
任何时候都可以运行4个模拟的建议?

fnatzsnv

fnatzsnv1#

在Ubuntu中安装moreutils软件包,然后使用parallel实用程序:

parallel -j 4 ./sim -r -- 1 2 3 4 5 6 7 8 ...
ioekq8ef

ioekq8ef2#

如果您不想安装parallel实用程序(假设它按指示工作,它看起来很整洁),那么您可以修改此Perl脚本(基本上,更改执行的命令),并可能减少监视:

#!/usr/bin/env perl
use strict;
use warnings;
use constant MAX_KIDS => 4;

$| = 1;

my %pids;
my $kids = 0;   # Number of kids

for my $i (1..20)
{
    my $pid;
    if (($pid = fork()) == 0)
    {
        my $tm = int(rand() * (10 - 2) + 2);
        print "sleep $tm\n";
        # Using exec in a block on its own is the documented way to
        # avoid the warning:
        # Statement unlikely to be reached at filename.pl line NN.
        #   (Maybe you meant system() when you said exec()?)
        # Yes, I know the print and exit statements should never be
        # reached, but, dammit, sometimes things go wrong!
        { exec "sleep", $tm; }
        print STDERR "Oops: couldn't sleep $tm!\n";
        exit 1;
    }
    $pids{$pid} = 1;
    $kids++;
    my $time = time;
    print "PID: $pid; Kids: $kids; Time: $time\n";
    if ($kids >= MAX_KIDS)
    {
        my $kid = waitpid(-1, 0);
        print "Kid: $kid ($?)\n";
        if ($kid != -1)
        {
            delete $pids{$kid};
            $kids--;
        }
    }
}

while ((my $kid = waitpid(-1, 0)) > 0)
{
    my $time = time;
    print "Kid: $kid (Status: $?); Time: $time\n";
    delete $pids{$kid};
    $kids--;
}

# This should not do anything - and doesn't (any more!).
foreach my $pid (keys %pids)
{
    printf "Undead: $pid\n";
}

输出示例:

PID: 20152; Kids: 1; Time: 1383436882
PID: 20153; Kids: 2; Time: 1383436882
sleep 5
PID: 20154; Kids: 3; Time: 1383436882
sleep 7
sleep 9
PID: 20155; Kids: 4; Time: 1383436882
sleep 4
Kid: 20155 (0)
PID: 20156; Kids: 4; Time: 1383436886
sleep 4
Kid: 20152 (0)
PID: 20157; Kids: 4; Time: 1383436887
sleep 2
Kid: 20153 (0)
PID: 20158; Kids: 4; Time: 1383436889
sleep 9
Kid: 20157 (0)
PID: 20159; Kids: 4; Time: 1383436889
sleep 6
Kid: 20156 (0)
PID: 20160; Kids: 4; Time: 1383436890
sleep 6
Kid: 20154 (0)
PID: 20161; Kids: 4; Time: 1383436891
sleep 9
Kid: 20159 (0)
PID: 20162; Kids: 4; Time: 1383436895
sleep 7
Kid: 20160 (0)
PID: 20163; Kids: 4; Time: 1383436896
sleep 9
Kid: 20158 (0)
PID: 20164; Kids: 4; Time: 1383436898
sleep 6
Kid: 20161 (0)
PID: 20165; Kids: 4; Time: 1383436900
sleep 9
Kid: 20162 (0)
PID: 20166; Kids: 4; Time: 1383436902
sleep 9
Kid: 20164 (0)
PID: 20167; Kids: 4; Time: 1383436904
sleep 2
Kid: 20163 (0)
PID: 20168; Kids: 4; Time: 1383436905
sleep 6
Kid: 20167 (0)
PID: 20169; Kids: 4; Time: 1383436906
sleep 9
Kid: 20165 (0)
PID: 20170; Kids: 4; Time: 1383436909
sleep 4
Kid: 20168 (0)
PID: 20171; Kids: 4; Time: 1383436911
Kid: 20166 (0)
sleep 9
Kid: 20170 (Status: 0); Time: 1383436913
Kid: 20169 (Status: 0); Time: 1383436915
Kid: 20171 (Status: 0); Time: 1383436920
wgeznvg7

wgeznvg73#

NUMJOBS=30
NUMPOOLS=4

seq 1 "$NUMJOBS" | for p in $(seq 1 $NUMPOOLS); do
    while read x; do ./sim -r "$x"; done &
done

for循环创建了一个后台进程池,它从共享的标准输入中读取数据以启动模拟。每个后台进程在其模拟运行时“阻塞”,然后从seq命令中读取下一个作业号。
如果没有for循环,它可能更容易理解:

seq 1 "$NUMJOBS" | {
    while read x; do ./sim -r "$x"; done &
    while read x; do ./sim -r "$x"; done &
    while read x; do ./sim -r "$x"; done &
    while read x; do ./sim -r "$x"; done &
}

假设sim花费非平凡的时间量来运行,第一个while将从其标准输入读取1,第二个2,等等。无论哪个sim首先完成,该while循环将从标准输入读取read 5;下一个要完成的将读取6,依此类推。一旦最后一个模拟开始,每个read将失败,导致循环退出。

相关问题