遵循最新的历史服务器文档https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/historyserver.html jobmanager.archive.fs.dir
可以在启动yarn会话集群后在hdfs上创建。
作业完成运行后,将在下面创建一个名为作业id的空文件 jobmanager.archive.fs.dir
.
历史服务器已启动并运行。
从历史服务器日志:
2020-09-24 22:39:43,270 DEBUG org.apache.flink.runtime.webmonitor.history.HistoryServerArchiveFetcher [] - Starting archive fetching.
2020-09-24 22:39:43,270 DEBUG org.apache.flink.runtime.webmonitor.history.HistoryServerArchiveFetcher [] - Checking archive directory hdfs://ltx1-holdemnn01.grid.linkedin.com:9000/user/cyzhang/flink-history-server.
2020-09-24 22:39:43,272 INFO org.apache.flink.runtime.webmonitor.history.HistoryServerArchiveFetcher [] - Processing archive hdfs://hostname/user/cyzhang/flink-history-server/2daf03dd7f9129637ced43d9237c1328.
2020-09-24 22:39:43,272 ERROR org.apache.flink.runtime.webmonitor.history.HistoryServerArchiveFetcher [] - Critical failure while fetching/processing job archives.
java.lang.NullPointerException: null
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:919) ~[hadoop-common-2.10.0.123.jar:?]
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:120) ~[flink-hadoop-fs-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.open(HadoopFileSystem.java:37) ~[flink-hadoop-fs-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.history.FsJobArchivist.getArchivedJsons(FsJobArchivist.java:108) ~[flink-runtime_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.webmonitor.history.HistoryServerArchiveFetcher$JobArchiveFetcherTask.run(HistoryServerArchiveFetcher.java:225) ~[flink-dist_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.util.Runnables.lambda$withUncaughtExceptionHandler$0(Runnables.java:40) ~[flink-runtime_2.11-1.11.1.jar:1.11.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_172]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_172]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_172]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_172]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
但是,上面的错误似乎是由于上一个作业日志为空,因为它试图监视以前完成的作业。
我想这里的主要问题是为什么作业将空日志发送到其hdfs jobmanager.archive.fs.dir
在它运行完之后。
我错过了一些配置设置吗?
暂无答案!
目前还没有任何答案,快来回答吧!