这是原始wordcount.java代码的一部分。
public static void main(String[] args) throws Exception {
// set up the execution environment
final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
// get input data
DataSet<String> text = env.fromElements(
"To be, or not to be,--that is the question:--",
"Whether 'tis nobler in the mind to suffer",
"The slings and arrows of outrageous fortune",
"Or to take arms against a sea of troubles,"
);
//DataSet<String> text = env.readTextFile("file:///home/jypark2/data3.txt");
DataSet<Tuple2<String, Integer>> counts =
// split up the lines in pairs (2-tuples) containing: (word,1)
text.flatMap(new LineSplitter())
// group by the tuple field "0" and sum up tuple field "1"
.groupBy(0)
.sum(1);
// execute and print result
counts.print();
}
我想从文本文件中读取,所以我修改了这个代码。
public static void main(String[] args) throws Exception {
// set up the execution environment
final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
// get input data
DataSet<String> text = env.readTextFile("file:///home/jypark2/data3.txt");
DataSet<Tuple2<String, Integer>> counts =
// split up the lines in pairs (2-tuples) containing: (word,1)
text.flatMap(new LineSplitter())
// group by the tuple field "0" and sum up tuple field "1"
.groupBy(0)
.sum(1);
// execute and print result
counts.print();
}
但是有一个运行时错误。但我解决不了这个问题。
在此处输入图像描述
为什么会发生这样的事?我该怎么解决?
1条答案
按热度按时间0sgqnhkj1#
如果在大规模并行设置(100多个并行线程)中运行flink,则需要通过配置参数taskmanager.network.numberofbuffers调整网络缓冲区的数量。根据经验,缓冲区的数量至少应为4NumberOfTaskManagernumberofslotspertaskmanager^2。有关详细信息,请参阅配置参考。
来自flink常见问题解答:https://flink.apache.org/faq.html#i-get-an-error-message-saying-that-not-enough-buffers-are-available-how-do-i-fix-这个