flink调用TargetException网络缓冲区数量不足

7y4bm7vi  于 2021-06-25  发布在  Flink
关注(0)|答案(1)|浏览(430)

这是原始wordcount.java代码的一部分。

  1. public static void main(String[] args) throws Exception {
  2. // set up the execution environment
  3. final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
  4. // get input data
  5. DataSet<String> text = env.fromElements(
  6. "To be, or not to be,--that is the question:--",
  7. "Whether 'tis nobler in the mind to suffer",
  8. "The slings and arrows of outrageous fortune",
  9. "Or to take arms against a sea of troubles,"
  10. );
  11. //DataSet<String> text = env.readTextFile("file:///home/jypark2/data3.txt");
  12. DataSet<Tuple2<String, Integer>> counts =
  13. // split up the lines in pairs (2-tuples) containing: (word,1)
  14. text.flatMap(new LineSplitter())
  15. // group by the tuple field "0" and sum up tuple field "1"
  16. .groupBy(0)
  17. .sum(1);
  18. // execute and print result
  19. counts.print();
  20. }

我想从文本文件中读取,所以我修改了这个代码。

  1. public static void main(String[] args) throws Exception {
  2. // set up the execution environment
  3. final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
  4. // get input data
  5. DataSet<String> text = env.readTextFile("file:///home/jypark2/data3.txt");
  6. DataSet<Tuple2<String, Integer>> counts =
  7. // split up the lines in pairs (2-tuples) containing: (word,1)
  8. text.flatMap(new LineSplitter())
  9. // group by the tuple field "0" and sum up tuple field "1"
  10. .groupBy(0)
  11. .sum(1);
  12. // execute and print result
  13. counts.print();
  14. }

但是有一个运行时错误。但我解决不了这个问题。
在此处输入图像描述
为什么会发生这样的事?我该怎么解决?

0sgqnhkj

0sgqnhkj1#

如果在大规模并行设置(100多个并行线程)中运行flink,则需要通过配置参数taskmanager.network.numberofbuffers调整网络缓冲区的数量。根据经验,缓冲区的数量至少应为4NumberOfTaskManagernumberofslotspertaskmanager^2。有关详细信息,请参阅配置参考。
来自flink常见问题解答:https://flink.apache.org/faq.html#i-get-an-error-message-saying-that-not-enough-buffers-are-available-how-do-i-fix-这个

相关问题