java—spring批处理的并行线程数上限为10

x6h2sr28  于 2021-08-25  发布在  Java
关注(0)|答案(1)|浏览(529)

我有以下任务执行器，用于运行带分区的10秒睡眠时间tasklet：

<bean id="taskExecutor" class="org.springframework.core.task.SimpleAsyncTaskExecutor">
        <property name="concurrencyLimit" value="${concurrency_limit}"/>
    </bean>
<batch:step id="stepWait">
        <batch:partition partitioner="partitionerWait">
            <batch:handler grid-size="1" task-executor="taskExecutor"/>
            <batch:step>
                <batch:tasklet ref="taskletWait"/>
            </batch:step>
        </batch:partition>
    </batch:step>
public class TaskletWait implements Tasklet {
    private final Logger logger = LoggerFactory.getLogger(TaskletWait.class);
   @Override
   public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext){
       String stepName = contribution.getStepExecution().getStepName();
       logger.info("starting step: " + stepName);
        try {
           TimeUnit.SECONDS.sleep(10);
        } catch (InterruptedException e) {
           logger.error(e.getMessage());
        }
        contribution.setExitStatus(ExitStatus.COMPLETED);
        return RepeatStatus.FINISHED;
   }
}

并行运行的线程数正在使用我在application.properties中设置的值进行正确调整，直到达到10。无论我设置10、15、25还是100，只有10个线程并行运行。以身作则 concurrency_limit=15 :

2021-07-06 13:31:34.937 INFO [SimpleAsyncTaskExecutor-12] [execute] starting step: stepWait:4 
2021-07-06 13:31:34.937 INFO [SimpleAsyncTaskExecutor-2] [execute] starting step: stepWait:30 
2021-07-06 13:31:34.937 INFO [SimpleAsyncTaskExecutor-1] [execute] starting step: stepWait:26 
2021-07-06 13:31:34.937 INFO [SimpleAsyncTaskExecutor-9] [execute] starting step: stepWait:17 
2021-07-06 13:31:34.937 INFO [SimpleAsyncTaskExecutor-3] [execute] starting step: stepWait:15 
2021-07-06 13:31:34.937 INFO [SimpleAsyncTaskExecutor-8] [execute] starting step: stepWait:11 
2021-07-06 13:31:34.939 INFO [SimpleAsyncTaskExecutor-10] [execute] starting step: stepWait:7 
2021-07-06 13:31:34.940 INFO [SimpleAsyncTaskExecutor-5] [execute] starting step: stepWait:1 
2021-07-06 13:31:34.941 INFO [SimpleAsyncTaskExecutor-15] [execute] starting step: stepWait:5 
2021-07-06 13:31:34.943 INFO [SimpleAsyncTaskExecutor-14] [execute] starting step: stepWait:23 
2021-07-06 13:31:44.941 INFO [SimpleAsyncTaskExecutor-6] [execute] starting step: stepWait:32 
2021-07-06 13:31:44.941 INFO [SimpleAsyncTaskExecutor-4] [execute] starting step: stepWait:19 
2021-07-06 13:31:44.941 INFO [SimpleAsyncTaskExecutor-7] [execute] starting step: stepWait:16 
2021-07-06 13:31:44.942 INFO [SimpleAsyncTaskExecutor-11] [execute] starting step: stepWait:13 
2021-07-06 13:31:44.943 INFO [SimpleAsyncTaskExecutor-13] [execute] starting step: stepWait:20 
2021-07-06 13:31:44.950 INFO [SimpleAsyncTaskExecutor-2] [execute] Step: [stepWait:30] executed in 10s21ms 
2021-07-06 13:31:44.950 INFO [SimpleAsyncTaskExecutor-8] [execute] Step: [stepWait:11] executed in 10s20ms 
2021-07-06 13:31:44.950 INFO [SimpleAsyncTaskExecutor-5] [execute] Step: [stepWait:1] executed in 10s21ms 
2021-07-06 13:31:44.950 INFO [SimpleAsyncTaskExecutor-9] [execute] Step: [stepWait:17] executed in 10s20ms 
2021-07-06 13:31:44.950 INFO [SimpleAsyncTaskExecutor-14] [execute] Step: [stepWait:23] executed in 10s18ms 
2021-07-06 13:31:44.950 INFO [SimpleAsyncTaskExecutor-10] [execute] Step: [stepWait:7] executed in 10s20ms 
2021-07-06 13:31:44.950 INFO [SimpleAsyncTaskExecutor-1] [execute] Step: [stepWait:26] executed in 10s22ms 
2021-07-06 13:31:44.951 INFO [SimpleAsyncTaskExecutor-3] [execute] Step: [stepWait:15] executed in 10s22ms 
2021-07-06 13:31:44.958 INFO [SimpleAsyncTaskExecutor-12] [execute] Step: [stepWait:4] executed in 10s29ms 
2021-07-06 13:31:44.959 INFO [SimpleAsyncTaskExecutor-15] [execute] Step: [stepWait:5] executed in 10s27ms 
2021-07-06 13:31:44.961 INFO [SimpleAsyncTaskExecutor-16] [execute] starting step: stepWait:6 
2021-07-06 13:31:44.962 INFO [SimpleAsyncTaskExecutor-17] [execute] starting step: stepWait:9 
2021-07-06 13:31:44.963 INFO [SimpleAsyncTaskExecutor-18] [execute] starting step: stepWait:24 
2021-07-06 13:31:44.963 INFO [SimpleAsyncTaskExecutor-19] [execute] starting step: stepWait:27 
2021-07-06 13:31:44.963 INFO [SimpleAsyncTaskExecutor-20] [execute] starting step: stepWait:3 
2021-07-06 13:31:54.944 INFO [SimpleAsyncTaskExecutor-21] [execute] starting step: stepWait:10 
2021-07-06 13:31:54.946 INFO [SimpleAsyncTaskExecutor-23] [execute] starting step: stepWait:33 
2021-07-06 13:31:54.947 INFO [SimpleAsyncTaskExecutor-22] [execute] starting step: stepWait:12 
2021-07-06 13:31:54.948 INFO [SimpleAsyncTaskExecutor-7] [execute] Step: [stepWait:16] executed in 20s19ms 
2021-07-06 13:31:54.951 INFO [SimpleAsyncTaskExecutor-13] [execute] Step: [stepWait:20] executed in 20s20ms 
2021-07-06 13:31:54.952 INFO [SimpleAsyncTaskExecutor-6] [execute] Step: [stepWait:32] executed in 20s22ms 
2021-07-06 13:31:54.954 INFO [SimpleAsyncTaskExecutor-24] [execute] starting step: stepWait:2 
2021-07-06 13:31:54.954 INFO [SimpleAsyncTaskExecutor-4] [execute] Step: [stepWait:19] executed in 20s25ms 
2021-07-06 13:31:54.955 INFO [SimpleAsyncTaskExecutor-11] [execute] Step: [stepWait:13] executed in 20s25ms 
2021-07-06 13:31:54.957 INFO [SimpleAsyncTaskExecutor-25] [execute] starting step: stepWait:22 
2021-07-06 13:31:54.967 INFO [SimpleAsyncTaskExecutor-16] [execute] Step: [stepWait:6] executed in 10s13ms 
2021-07-06 13:31:54.968 INFO [SimpleAsyncTaskExecutor-17] [execute] Step: [stepWait:9] executed in 10s13ms 
[...]

第11到第15个线程似乎已启动，但等待一个线程结束后才能实际执行，可以看到它们的执行时间为20秒。
当前日志是在windows上使用java 11生成的。我试过了 org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor 而不是 SimpleAsyncTaskExecutor 但同样的结果。我在windows和linux上都试过，但结果相同
是什么导致了这种限制？如何使其并行运行10多个线程？
编辑：大多数情况下，只有10次并行运行，但有时只有9次，这似乎是流畅的。
edit2：我正在添加concurrencythrottlesupport的调试日志

2021-07-06 17:24:04.860 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 0 
2021-07-06 17:24:04.860 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 1 
2021-07-06 17:24:04.861 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 2 
2021-07-06 17:24:04.861 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 3 
2021-07-06 17:24:04.861 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 4 
2021-07-06 17:24:04.862 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 5 
2021-07-06 17:24:04.864 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 6 
2021-07-06 17:24:04.864 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 7 
2021-07-06 17:24:04.865 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 8 
2021-07-06 17:24:04.865 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 9 
2021-07-06 17:24:04.866 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 10 
2021-07-06 17:24:04.866 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 11 
2021-07-06 17:24:04.866 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 12 
2021-07-06 17:24:04.867 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 13 
2021-07-06 17:24:04.867 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 14 
2021-07-06 17:24:04.868 DEBUG [main] [beforeAccess] Concurrency count 15 has reached limit 15 - blocking 
2021-07-06 17:24:04.869 INFO [SimpleAsyncTaskExecutor-9] [execute] starting step: wait:3 
2021-07-06 17:24:04.869 INFO [SimpleAsyncTaskExecutor-7] [execute] starting step: wait:7 
2021-07-06 17:24:04.869 INFO [SimpleAsyncTaskExecutor-2] [execute] starting step: wait:11 
2021-07-06 17:24:04.869 INFO [SimpleAsyncTaskExecutor-3] [execute] starting step: wait:2 
2021-07-06 17:24:04.869 INFO [SimpleAsyncTaskExecutor-5] [execute] starting step: wait:16 
2021-07-06 17:24:04.869 INFO [SimpleAsyncTaskExecutor-12] [execute] starting step: wait:4 
2021-07-06 17:24:04.869 INFO [SimpleAsyncTaskExecutor-10] [execute] starting step: wait:34 
2021-07-06 17:24:04.869 INFO [SimpleAsyncTaskExecutor-11] [execute] starting step: wait:27 
2021-07-06 17:24:04.869 INFO [SimpleAsyncTaskExecutor-4] [execute] starting step: wait:18 
2021-07-06 17:24:04.869 INFO [SimpleAsyncTaskExecutor-1] [execute] starting step: wait:29 
2021-07-06 17:24:14.873 INFO [SimpleAsyncTaskExecutor-6] [execute] starting step: wait:33 
2021-07-06 17:24:14.873 INFO [SimpleAsyncTaskExecutor-13] [execute] starting step: wait:0 
2021-07-06 17:24:14.875 INFO [SimpleAsyncTaskExecutor-15] [execute] starting step: wait:25 
2021-07-06 17:24:14.875 INFO [SimpleAsyncTaskExecutor-14] [execute] starting step: wait:13 
2021-07-06 17:24:14.882 INFO [SimpleAsyncTaskExecutor-8] [execute] starting step: wait:6 
2021-07-06 17:24:14.885 INFO [SimpleAsyncTaskExecutor-3] [execute] Step: [wait:2] executed in 10s23ms 
2021-07-06 17:24:14.885 INFO [SimpleAsyncTaskExecutor-11] [execute] Step: [wait:27] executed in 10s18ms 
2021-07-06 17:24:14.885 INFO [SimpleAsyncTaskExecutor-5] [execute] Step: [wait:16] executed in 10s22ms 
2021-07-06 17:24:14.885 INFO [SimpleAsyncTaskExecutor-9] [execute] Step: [wait:3] executed in 10s19ms 
2021-07-06 17:24:14.885 INFO [SimpleAsyncTaskExecutor-12] [execute] Step: [wait:4] executed in 10s18ms 
2021-07-06 17:24:14.885 INFO [SimpleAsyncTaskExecutor-10] [execute] Step: [wait:34] executed in 10s19ms 
2021-07-06 17:24:14.885 INFO [SimpleAsyncTaskExecutor-7] [execute] Step: [wait:7] executed in 10s20ms 
2021-07-06 17:24:14.885 INFO [SimpleAsyncTaskExecutor-2] [execute] Step: [wait:11] executed in 10s24ms 
2021-07-06 17:24:14.885 INFO [SimpleAsyncTaskExecutor-1] [execute] Step: [wait:29] executed in 10s25ms 
2021-07-06 17:24:14.886 DEBUG [SimpleAsyncTaskExecutor-3] [afterAccess] Returning from throttle at concurrency count 14 
2021-07-06 17:24:14.887 DEBUG [SimpleAsyncTaskExecutor-12] [afterAccess] Returning from throttle at concurrency count 13 
2021-07-06 17:24:14.887 DEBUG [SimpleAsyncTaskExecutor-9] [afterAccess] Returning from throttle at concurrency count 12 
2021-07-06 17:24:14.887 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 12 
2021-07-06 17:24:14.888 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 13 
2021-07-06 17:24:14.888 DEBUG [SimpleAsyncTaskExecutor-10] [afterAccess] Returning from throttle at concurrency count 13 
2021-07-06 17:24:14.888 DEBUG [SimpleAsyncTaskExecutor-5] [afterAccess] Returning from throttle at concurrency count 12 
2021-07-06 17:24:14.888 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 12 
2021-07-06 17:24:14.889 DEBUG [SimpleAsyncTaskExecutor-7] [afterAccess] Returning from throttle at concurrency count 12 
2021-07-06 17:24:14.889 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 12 
2021-07-06 17:24:14.889 DEBUG [SimpleAsyncTaskExecutor-2] [afterAccess] Returning from throttle at concurrency count 12 
2021-07-06 17:24:14.890 DEBUG [SimpleAsyncTaskExecutor-11] [afterAccess] Returning from throttle at concurrency count 11 
2021-07-06 17:24:14.890 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 11 
2021-07-06 17:24:14.890 DEBUG [SimpleAsyncTaskExecutor-1] [afterAccess] Returning from throttle at concurrency count 11 
2021-07-06 17:24:14.890 INFO [SimpleAsyncTaskExecutor-4] [execute] Step: [wait:18] executed in 10s29ms 
2021-07-06 17:24:14.890 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 11 
2021-07-06 17:24:14.890 INFO [SimpleAsyncTaskExecutor-16] [execute] starting step: wait:30 
2021-07-06 17:24:14.891 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 12 
2021-07-06 17:24:14.892 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 13 
2021-07-06 17:24:14.892 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 14 
2021-07-06 17:24:14.892 INFO [SimpleAsyncTaskExecutor-17] [execute] starting step: wait:12 
2021-07-06 17:24:14.892 DEBUG [main] [beforeAccess] Concurrency count 15 has reached limit 15 - blocking 
2021-07-06 17:24:14.893 DEBUG [SimpleAsyncTaskExecutor-4] [afterAccess] Returning from throttle at concurrency count 14 
2021-07-06 17:24:14.893 INFO [SimpleAsyncTaskExecutor-18] [execute] starting step: wait:28 
2021-07-06 17:24:14.893 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 14 
2021-07-06 17:24:14.894 DEBUG [main] [beforeAccess] Concurrency count 15 has reached limit 15 - blocking 
2021-07-06 17:24:14.894 INFO [SimpleAsyncTaskExecutor-19] [execute] starting step: wait:24 
2021-07-06 17:24:14.899 INFO [SimpleAsyncTaskExecutor-21] [execute] starting step: wait:23 
2021-07-06 17:24:24.878 INFO [SimpleAsyncTaskExecutor-20] [execute] starting step: wait:1 
2021-07-06 17:24:24.879 INFO [SimpleAsyncTaskExecutor-22] [execute] starting step: wait:32 
2021-07-06 17:24:24.879 INFO [SimpleAsyncTaskExecutor-23] [execute] starting step: wait:15 
2021-07-06 17:24:24.879 INFO [SimpleAsyncTaskExecutor-24] [execute] starting step: wait:10 
2021-07-06 17:24:24.884 INFO [SimpleAsyncTaskExecutor-6] [execute] Step: [wait:33] executed in 20s19ms 
2021-07-06 17:24:24.884 INFO [SimpleAsyncTaskExecutor-13] [execute] Step: [wait:0] executed in 20s17ms 
2021-07-06 17:24:24.885 INFO [SimpleAsyncTaskExecutor-15] [execute] Step: [wait:25] executed in 20s17ms 
2021-07-06 17:24:24.885 INFO [SimpleAsyncTaskExecutor-25] [execute] starting step: wait:17 
2021-07-06 17:24:24.894 INFO [SimpleAsyncTaskExecutor-14] [execute] Step: [wait:13] executed in 20s26ms 
2021-07-06 17:24:24.894 INFO [SimpleAsyncTaskExecutor-8] [execute] Step: [wait:6] executed in 20s29ms 
2021-07-06 17:24:24.897 DEBUG [SimpleAsyncTaskExecutor-6] [afterAccess] Returning from throttle at concurrency count 14 
2021-07-06 17:24:24.898 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 14 
2021-07-06 17:24:24.898 DEBUG [SimpleAsyncTaskExecutor-13] [afterAccess] Returning from throttle at concurrency count 14 
2021-07-06 17:24:24.899 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 14 
2021-07-06 17:24:24.899 INFO [SimpleAsyncTaskExecutor-16] [execute] Step: [wait:30] executed in 10s12ms 
2021-07-06 17:24:24.899 DEBUG [SimpleAsyncTaskExecutor-15] [afterAccess] Returning from throttle at concurrency count 14 
2021-07-06 17:24:24.899 DEBUG [main] [beforeAccess] Entering throttle at concurrency count 14 
2021-07-06 17:24:24.900 INFO [SimpleAsyncTaskExecutor-17] [execute] Step: [wait:12] executed in 10s12ms
Javaspring-bootmultithreadingspring-batch

1条答案

按热度按时间
pdtvr36n

pdtvr36n1#

事实上，我不复制您的代码样本。下面是一个重现错误的示例：github.com/alexandrechi/spring batch lab/tree/main/issues/…。与代码示例的唯一区别是注解springbootapplication而不是配置。
所以我得出结论，这与通过引导进行自动配置有关。
我想我已经接近了，将enableautoconfiguration和实现“com.h2database:h2:1.4.198”联系起来产生了问题。它似乎创建了一个自动连接的数据源，同时连接的数量有限
事实上，我在日志中看到：

DEBUG com.zaxxer.hikari.HikariConfig - maximumPoolSize.................10

这是使用hikaricp的最大池大小的默认值。这意味着15个线程不能获得超过10个数据库连接，因此一次只能并行运行其中的10个。
通过添加以下属性将最大池大小增加到15可以解决此问题：

spring.datasource.hikari.maximum-pool-size=15

我相应地更新了回购协议，以便您可以检查结果。

赞(0）分享  回复(0）举报  2021-08-25
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com