flink调用TargetException网络缓冲区数量不足

7y4bm7vi  于 2021-06-25  发布在  Flink
关注(0)|答案(1)|浏览(422)

这是原始wordcount.java代码的一部分。

public static void main(String[] args) throws Exception {

    // set up the execution environment
    final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();

    // get input data
    DataSet<String> text = env.fromElements(
            "To be, or not to be,--that is the question:--",
            "Whether 'tis nobler in the mind to suffer",
            "The slings and arrows of outrageous fortune",
            "Or to take arms against a sea of troubles,"
            );
    //DataSet<String> text = env.readTextFile("file:///home/jypark2/data3.txt");

    DataSet<Tuple2<String, Integer>> counts =
            // split up the lines in pairs (2-tuples) containing: (word,1)
            text.flatMap(new LineSplitter())
            // group by the tuple field "0" and sum up tuple field "1"
            .groupBy(0)
            .sum(1);

    // execute and print result
    counts.print();

}

我想从文本文件中读取,所以我修改了这个代码。

public static void main(String[] args) throws Exception {

    // set up the execution environment
    final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();

    // get input data

    DataSet<String> text = env.readTextFile("file:///home/jypark2/data3.txt");

    DataSet<Tuple2<String, Integer>> counts =
            // split up the lines in pairs (2-tuples) containing: (word,1)
            text.flatMap(new LineSplitter())
            // group by the tuple field "0" and sum up tuple field "1"
            .groupBy(0)
            .sum(1);

    // execute and print result
    counts.print();

}

但是有一个运行时错误。但我解决不了这个问题。
在此处输入图像描述
为什么会发生这样的事?我该怎么解决?

0sgqnhkj

0sgqnhkj1#

如果在大规模并行设置(100多个并行线程)中运行flink,则需要通过配置参数taskmanager.network.numberofbuffers调整网络缓冲区的数量。根据经验,缓冲区的数量至少应为4NumberOfTaskManagernumberofslotspertaskmanager^2。有关详细信息,请参阅配置参考。
来自flink常见问题解答:https://flink.apache.org/faq.html#i-get-an-error-message-saying-that-not-enough-buffers-are-available-how-do-i-fix-这个

相关问题