DataX java.lang.OutOfMemoryError: Failed to create a thread

0md85ypi  于 3个月前  发布在  Java
关注(0)|答案(1)|浏览(86)

在16C32G配置的虚机服务器上,通过java应用维护了并行度为9的线程池,线程池中总共有63个任务,并行度为9代表同时会启动9个datax脚本,脚本为: python /app/datax/datax/bin/datax.py ...
当线程池中的任务执行完一半时,剩下的正在执行的任务抛出日志中抛出如下错误,但是此时服务器的内存剩余充足(如下所示),datax.py的JVM参数设置: DEFAULT_JVM = "-Xms256M -Xmx512M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=%s/log" % (DATAX_HOME) ,通过各自方式排查,均找不到解决方案,请大佬分析一下该问题原因所在,多谢。

$ free -m
             total       used       free     shared    buffers     cached
Mem:         32097      12850      19247          1        806       7877
-/+ buffers/cache:       4166      27930
Swap:         8199         49       8150

json文件的setting属性配置参数如下:

"setting": {
            "speed": {
                "channel": "10",
				"record":-1,
				"byte":-1,
				"batchSize":4000
            }
        }

日志错误信息:
2023-03-17 09:54:25.127 [job-0] INFO JobContainer - jobContainer starts to do prepare ...
2023-03-17 09:54:25.127 [job-0] INFO JobContainer - DataX Reader.Job [oraclereader] do prepare work .
2023-03-17 09:54:25.128 [job-0] INFO JobContainer - DataX Writer.Job [oraclewriter] do prepare work .
2023-03-17 09:54:25.129 [job-0] INFO JobContainer - jobContainer starts to do split ...
2023-03-17 09:54:25.130 [job-0] INFO JobContainer - Job set Channel-Number to 10 channels.
2023-03-17 09:54:25.134 [job-0] INFO JobContainer - DataX Reader.Job [oraclereader] splits to [1] tasks.
2023-03-17 09:54:25.135 [job-0] INFO JobContainer - DataX Writer.Job [oraclewriter] splits to [1] tasks.
2023-03-17 09:54:25.165 [job-0] INFO JobContainer - jobContainer starts to do schedule ...
2023-03-17 09:54:25.176 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2023-03-17 09:54:25.178 [job-0] INFO JobContainer - Running by standalone Mode.
2023-03-17 09:54:25.381 [job-0] ERROR JobContainer - Exception when job run
java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 11
at java.lang.Thread.start(Thread.java:988) ~[na:2.9 (11-06-2019)]
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:968) ~[na:1.8.0]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378) ~[na:1.8.0]
at com.alibaba.datax.core.job.scheduler.processinner.ProcessInnerScheduler.startAllTaskGroup(ProcessInnerScheduler.java:30) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.job.scheduler.AbstractScheduler.schedule(AbstractScheduler.java:55) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.job.JobContainer.schedule(JobContainer.java:535) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:119) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.Engine.start(Engine.java:92) [datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.Engine.entry(Engine.java:171) [datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.Engine.main(Engine.java:204) [datax-core-0.0.1-SNAPSHOT.jar:na]
2023-03-17 09:54:25.404 [job-0] INFO StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00%

zbwhf8kr

zbwhf8kr1#

The same problem arises, how did you solve it

相关问题