flink作业部署

0ejtzxu1  于 2021-06-24  发布在  Flink
关注(0)|答案(2)|浏览(1007)

我试图在kubernetes集群（azureaks）中部署一个flink作业。作业群集刚启动就被中止，但任务管理器运行正常。
docker映像创建成功，没有任何异常。我能够运行docker映像以及ssh到docker映像。
我遵循了以下链接中提到的步骤：
https://github.com/apache/flink/tree/release-1.9/flink-container/kubernetes
在创建图像时，我提供了jobjar，它被复制到图像中的“/opt/artifacts”上。但仍然不明白为什么在作业集群pod日志中出现以下异常。

  1. Caused by: org.apache.flink.util.FlinkException: Failed to find job JAR on class path. Please provide the job class name explicitly.

我是kubernetes的新手，你能给我一些线索来调试这个问题吗。
请查看以下完整日志：
答。flink作业群集日志

  1. develk@ACIDLAELKV01:~/cntx_eng$ kubectl logs flink-job-cluster-kszwf
  2. Starting the job-cluster
  3. Starting standalonejob as a console application on host flink-job-cluster-kszwf.
  4. 2019-12-12 10:37:17,170 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - --------------------------------------------------------------------------------
  5. 2019-12-12 10:37:17,172 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Starting StandaloneJobClusterEntryPoint (Version: 1.8.0, Rev:4caec0d, Date:03.04.2019 @ 13:25:54 PDT)
  6. 2019-12-12 10:37:17,172 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  OS current user: flink
  7. 2019-12-12 10:37:17,173 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Current Hadoop/Kerberos user: <no hadoop dependency found>
  8. 2019-12-12 10:37:17,173 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  JVM: OpenJDK 64-Bit Server VM - IcedTea - 1.8/25.212-b04
  9. 2019-12-12 10:37:17,173 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Maximum heap size: 989 MiBytes
  10. 2019-12-12 10:37:17,173 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  JAVA_HOME: /usr/lib/jvm/java-1.8-openjdk/jre
  11. 2019-12-12 10:37:17,174 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  No Hadoop Dependency available
  12. 2019-12-12 10:37:17,174 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  JVM Options:
  13. 2019-12-12 10:37:17,174 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Xms1024m
  14. 2019-12-12 10:37:17,174 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Xmx1024m
  15. 2019-12-12 10:37:17,174 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Dlog4j.configuration=file:/opt/flink-1.8.0/conf/log4j-console.properties
  16. 2019-12-12 10:37:17,175 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Dlogback.configurationFile=file:/opt/flink-1.8.0/conf/logback-console.xml
  17. 2019-12-12 10:37:17,175 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Program Arguments:
  18. 2019-12-12 10:37:17,175 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     --configDir
  19. 2019-12-12 10:37:17,175 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     /opt/flink-1.8.0/conf
  20. 2019-12-12 10:37:17,175 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Djobmanager.rpc.address=flink-job-cluster
  21. 2019-12-12 10:37:17,175 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Dparallelism.default=1
  22. 2019-12-12 10:37:17,176 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Dblob.server.port=6124
  23. 2019-12-12 10:37:17,176 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Dqueryable-state.server.ports=6125
  24. 2019-12-12 10:37:17,176 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Classpath: /opt/flink-1.8.0/lib/log4j-1.2.17.jar:/opt/flink-1.8.0/lib/slf4j-log4j12-1.7.15.jar:/opt/flink-1.8.0/lib/flink-dist_2.11-1.8.0.jar:::
  25. 2019-12-12 10:37:17,176 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - --------------------------------------------------------------------------------
  26. 2019-12-12 10:37:17,178 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Registered UNIX signal handlers for [TERM, HUP, INT]
  27. 2019-12-12 10:37:17,306 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.address, localhost
  28. 2019-12-12 10:37:17,306 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.port, 6123
  29. 2019-12-12 10:37:17,307 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.size, 1024m
  30. 2019-12-12 10:37:17,307 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.heap.size, 1024m
  31. 2019-12-12 10:37:17,307 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 1
  32. 2019-12-12 10:37:17,307 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: parallelism.default, 1
  33. 2019-12-12 10:37:17,336 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Starting StandaloneJobClusterEntryPoint.
  34. 2019-12-12 10:37:17,336 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Install default filesystem.
  35. 2019-12-12 10:37:17,343 INFO  org.apache.flink.core.fs.FileSystem                           - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available.
  36. 2019-12-12 10:37:17,352 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Install security context.
  37. 2019-12-12 10:37:17,362 INFO  org.apache.flink.runtime.security.modules.HadoopModuleFactory  - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath.
  38. 2019-12-12 10:37:17,381 INFO  org.apache.flink.runtime.security.SecurityUtils               - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath.
  39. 2019-12-12 10:37:17,382 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Initializing cluster services.
  40. 2019-12-12 10:37:17,638 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils         - Trying to start actor system at flink-job-cluster:6123
  41. 2019-12-12 10:37:18,163 INFO  akka.event.slf4j.Slf4jLogger                                  - Slf4jLogger started
  42. 2019-12-12 10:37:18,237 INFO  akka.remote.Remoting                                          - Starting remoting
  43. 2019-12-12 10:37:18,366 INFO  akka.remote.Remoting                                          - Remoting started; listening on addresses :[akka.tcp://flink@flink-job-cluster:6123]
  44. 2019-12-12 10:37:18,375 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils         - Actor system started at akka.tcp://flink@flink-job-cluster:6123
  45. 2019-12-12 10:37:18,398 INFO  org.apache.flink.configuration.Configuration                  - Config uses fallback configuration key 'jobmanager.rpc.address' instead of key 'rest.address'
  46. 2019-12-12 10:37:18,407 INFO  org.apache.flink.runtime.blob.BlobServer                      - Created BLOB server storage directory /tmp/blobStore-63338044-67c1-4872-a3d9-c94563b3a7c3
  47. 2019-12-12 10:37:18,412 INFO  org.apache.flink.runtime.blob.BlobServer                      - Started BLOB server at 0.0.0.0:6124 - max concurrent requests: 50 - max backlog: 1000
  48. 2019-12-12 10:37:18,428 INFO  org.apache.flink.runtime.metrics.MetricRegistryImpl           - No metrics reporter configured, no metrics will be exposed/reported.
  49. 2019-12-12 10:37:18,430 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Trying to start actor system at flink-job-cluster:0
  50. 2019-12-12 10:37:18,464 INFO  akka.event.slf4j.Slf4jLogger                                  - Slf4jLogger started
  51. 2019-12-12 10:37:18,472 INFO  akka.remote.Remoting                                          - Starting remoting
  52. 2019-12-12 10:37:18,480 INFO  akka.remote.Remoting                                          - Remoting started; listening on addresses :[akka.tcp://flink-metrics@flink-job-cluster:33529]
  53. 2019-12-12 10:37:18,482 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Actor system started at akka.tcp://flink-metrics@flink-job-cluster:33529
  54. 2019-12-12 10:37:18,490 INFO  org.apache.flink.runtime.blob.TransientBlobCache              - Created BLOB cache storage directory /tmp/blobStore-ba64dcdb-5095-41fc-9c98-0f1528d95c40
  55. 2019-12-12 10:37:18,514 INFO  org.apache.flink.configuration.Configuration                  - Config uses fallback configuration key 'jobmanager.rpc.address' instead of key 'rest.address'
  56. 2019-12-12 10:37:18,515 WARN  org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint  - Upload directory /tmp/flink-web-f6be0c2d-5099-4bd6-bc72-a0ae1fc6448e/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available.
  57. 2019-12-12 10:37:18,516 INFO  org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint  - Created directory /tmp/flink-web-f6be0c2d-5099-4bd6-bc72-a0ae1fc6448e/flink-web-upload for file uploads.
  58. 2019-12-12 10:37:18,603 INFO  org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint  - Starting rest endpoint.
  59. 2019-12-12 10:37:18,872 WARN  org.apache.flink.runtime.webmonitor.WebMonitorUtils           - Log file environment variable 'log.file' is not set.
  60. 2019-12-12 10:37:18,872 WARN  org.apache.flink.runtime.webmonitor.WebMonitorUtils           - JobManager log files are unavailable in the web dashboard. Log file location not found in environment variable 'log.file' or configuration key 'Key: 'web.log.path' , default: null (fallback keys: [{key=jobmanager.web.log.path, isDeprecated=true}])'.
  61. 2019-12-12 10:37:19,115 INFO  org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint  - Rest endpoint listening at flink-job-cluster:8081
  62. 2019-12-12 10:37:19,116 INFO  org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint  - http://flink-job-cluster:8081 was granted leadership with leaderSessionID=00000000-0000-0000-0000-000000000000
  63. 2019-12-12 10:37:19,116 INFO  org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint  - Web frontend listening at http://flink-job-cluster:8081.
  64. 2019-12-12 10:37:19,239 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at akka://flink/user/resourcemanager .
  65. 2019-12-12 10:37:19,262 INFO  org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever  - Scanning class path for job JAR
  66. 2019-12-12 10:37:19,270 INFO  org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint  - Shutting down rest endpoint.
  67. 2019-12-12 10:37:19,295 INFO  org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint  - Removing cache directory /tmp/flink-web-f6be0c2d-5099-4bd6-bc72-a0ae1fc6448e/flink-web-ui
  68. 2019-12-12 10:37:19,299 INFO  org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint  - http://flink-job-cluster:8081 lost leadership
  69. 2019-12-12 10:37:19,299 INFO  org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint  - Shut down complete.
  70. 2019-12-12 10:37:19,302 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Shutting StandaloneJobClusterEntryPoint down with application status FAILED. Diagnostics org.apache.flink.util.FlinkException: Could not create the DispatcherResourceManagerComponent.
  71.         at org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResourceManagerComponentFactory.java:257)
  72.         at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:224)
  73.         at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:172)
  74.         at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
  75.         at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:171)
  76.         at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:535)
  77.         at org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.main(StandaloneJobClusterEntryPoint.java:105)
  78. Caused by: org.apache.flink.util.FlinkException: Failed to find job JAR on class path. Please provide the job class name explicitly.
  79.         at org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever.getJobClassNameOrScanClassPath(ClassPathJobGraphRetriever.java:131)
  80.         at org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever.createPackagedProgram(ClassPathJobGraphRetriever.java:114)
  81.         at org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever.retrieveJobGraph(ClassPathJobGraphRetriever.java:96)
  82.         at org.apache.flink.runtime.dispatcher.JobDispatcherFactory.createDispatcher(JobDispatcherFactory.java:62)
  83.         at org.apache.flink.runtime.dispatcher.JobDispatcherFactory.createDispatcher(JobDispatcherFactory.java:41)
  84.         at org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResourceManagerComponentFactory.java:184)
  85.         ... 6 more
  86. Caused by: java.util.NoSuchElementException: No JAR with manifest attribute for entry class
  87.         at org.apache.flink.container.entrypoint.JarManifestParser.findOnlyEntryClass(JarManifestParser.java:80)
  88.         at org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever.scanClassPathForJobJar(ClassPathJobGraphRetriever.java:137)
  89.         at org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever.getJobClassNameOrScanClassPath(ClassPathJobGraphRetriever.java:129)
  90.         ... 11 more
  91. .
  92. 2019-12-12 10:37:19,305 INFO  org.apache.flink.runtime.blob.BlobServer                      - Stopped BLOB server at 0.0.0.0:6124
  93. 2019-12-12 10:37:19,305 INFO  org.apache.flink.runtime.blob.TransientBlobCache              - Shutting down BLOB cache
  94. 2019-12-12 10:37:19,315 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Stopping Akka RPC service.
  95. 2019-12-12 10:37:19,320 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator         - Shutting down remote daemon.
  96. 2019-12-12 10:37:19,321 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator         - Shutting down remote daemon.
  97. 2019-12-12 10:37:19,323 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator         - Remote daemon shut down; proceeding with flushing remote transports.
  98. 2019-12-12 10:37:19,325 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator         - Remote daemon shut down; proceeding with flushing remote transports.
  99. 2019-12-12 10:37:19,354 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator         - Remoting shut down.
  100. 2019-12-12 10:37:19,356 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator         - Remoting shut down.
  101. 2019-12-12 10:37:19,378 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Stopped Akka RPC service.
  102. 2019-12-12 10:37:19,382 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Could not start cluster entrypoint StandaloneJobClusterEntryPoint.
  103. org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to initialize the cluster entrypoint StandaloneJobClusterEntryPoint.
  104.         at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:190)
  105.         at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:535)
  106.         at org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.main(StandaloneJobClusterEntryPoint.java:105)
  107. Caused by: org.apache.flink.util.FlinkException: Could not create the DispatcherResourceManagerComponent.
  108.         at org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResourceManagerComponentFactory.java:257)
  109.         at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:224)
  110.         at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:172)
  111.         at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
  112.         at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:171)
  113.         ... 2 more
  114. Caused by: org.apache.flink.util.FlinkException: Failed to find job JAR on class path. Please provide the job class name explicitly.
  115.         at org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever.getJobClassNameOrScanClassPath(ClassPathJobGraphRetriever.java:131)
  116.         at org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever.createPackagedProgram(ClassPathJobGraphRetriever.java:114)
  117.         at org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever.retrieveJobGraph(ClassPathJobGraphRetriever.java:96)
  118.         at org.apache.flink.runtime.dispatcher.JobDispatcherFactory.createDispatcher(JobDispatcherFactory.java:62)
  119.         at org.apache.flink.runtime.dispatcher.JobDispatcherFactory.createDispatcher(JobDispatcherFactory.java:41)
  120.         at org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResourceManagerComponentFactory.java:184)
  121.         ... 6 more
  122. Caused by: java.util.NoSuchElementException: No JAR with manifest attribute for entry class
  123.         at org.apache.flink.container.entrypoint.JarManifestParser.findOnlyEntryClass(JarManifestParser.java:80)
  124.         at org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever.scanClassPathForJobJar(ClassPathJobGraphRetriever.java:137)
  125.         at org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever.getJobClassNameOrScanClassPath(ClassPathJobGraphRetriever.java:129)
  126.         ... 11 more
  127. develk@ACIDLAELKV01:~/cntx_eng$

现在，我在“job cluster job.yaml.template”文件的参数部分添加了作业类名。
如下所示：

  1. args: ["job-cluster", 
  2.                "--job-classname", "com.flink.wordCountSimple",
  3.                "-Djobmanager.rpc.address=flink-job-cluster",

但在那之后我得到了以下例外：

  1. Caused by: org.apache.flink.util.FlinkException: Could not load the provided entrypoint class.

请参阅下面的详细日志。

  1. 2019-12-13 19:08:34,323 INFO  org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint  - Shut down complete.
  2. 2019-12-13 19:08:34,329 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Shutting StandaloneJobClusterEntryPoint down with application status FAILED. Diagnostics org.apache.flink.util.FlinkException: Could not create the DispatcherResourceManagerComponent.
  3.         at org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResourceManagerComponentFactory.java:257)
  4.         at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:224)
  5.         at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:172)
  6.         at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
  7.         at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:171)
  8.         at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:535)
  9.         at org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.main(StandaloneJobClusterEntryPoint.java:105)
  10. Caused by: org.apache.flink.util.FlinkException: Could not load the provided entrypoint class.
  11.         at org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever.createPackagedProgram(ClassPathJobGraphRetriever.java:119)
  12.         at org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever.retrieveJobGraph(ClassPathJobGraphRetriever.java:96)
  13.         at org.apache.flink.runtime.dispatcher.JobDispatcherFactory.createDispatcher(JobDispatcherFactory.java:62)
  14.         at org.apache.flink.runtime.dispatcher.JobDispatcherFactory.createDispatcher(JobDispatcherFactory.java:41)
  15.         at org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResourceManagerComponentFactory.java:184)
  16.         ... 6 more
  17. Caused by: java.lang.ClassNotFoundException: com.flink.wordCountSimple
  18.         at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
  19.         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  20.         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
  21.         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  22.         at org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever.createPackagedProgram(ClassPathJobGraphRetriever.java:116)
  23.         ... 10 more
  24. .
  25. 2019-12-13 19:08:34,337 INFO  org.apache.flink.runtime.blob.BlobServer                      - Stopped BLOB server at 0.0.0.0:6124
  26. 2019-12-13 19:08:34,338 INFO  org.apache.flink.runtime.blob.TransientBlobCache              - Shutting down BLOB cache
  27. 2019-12-13 19:08:34,364 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Stopping Akka RPC service.
  28. 2019-12-13 19:08:34,368 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator         - Shutting down remote daemon.
  29. 2019-12-13 19:08:34,372 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator         - Remote daemon shut down; proceeding with flushing remote transports.
  30. 2019-12-13 19:08:34,392 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator         - Shutting down remote daemon.
  31. 2019-12-13 19:08:34,392 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator         - Remote daemon shut down; proceeding with flushing remote transports.
  32. 2019-12-13 19:08:34,406 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator         - Remoting shut down.
  33. 2019-12-13 19:08:34,410 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator         - Remoting shut down.
  34. 2019-12-13 19:08:34,434 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Stopped Akka RPC service.
  35. 2019-12-13 19:08:34,443 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Could not start cluster entrypoint StandaloneJobClusterEntryPoint.
  36. org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to initialize the cluster entrypoint StandaloneJobClusterEntryPoint.
  37.         at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:190)
  38.         at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:535)
  39.         at org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.main(StandaloneJobClusterEntryPoint.java:105)
  40. Caused by: org.apache.flink.util.FlinkException: Could not create the DispatcherResourceManagerComponent.
  41.         at org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResourceManagerComponentFactory.java:257)
  42.         at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:224)
  43.         at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:172)
  44.         at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
  45.         at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:171)
  46.         ... 2 more
  47. Caused by: org.apache.flink.util.FlinkException: Could not load the provided entrypoint class.
  48.         at org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever.createPackagedProgram(ClassPathJobGraphRetriever.java:119)
  49.         at org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever.retrieveJobGraph(ClassPathJobGraphRetriever.java:96)
  50.         at org.apache.flink.runtime.dispatcher.JobDispatcherFactory.createDispatcher(JobDispatcherFactory.java:62)
  51.         at org.apache.flink.runtime.dispatcher.JobDispatcherFactory.createDispatcher(JobDispatcherFactory.java:41)
  52.         at org.apache.flink.runtime.entrypoint.component.AbstractDispatcherResourceManagerComponentFactory.create(AbstractDispatcherResourceManagerComponentFactory.java:184)
  53.         ... 6 more
  54. Caused by: java.lang.ClassNotFoundException: com.flink.wordCountSimple
  55.         at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
  56.         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  57.         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
  58.         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  59.         at org.apache.flink.container.entrypoint.ClassPathJobGraphRetriever.createPackagedProgram(ClassPathJobGraphRetriever.java:116)
  60.         ... 10 more
Azureapache-flinkkubernetesazure-aksdevops

2条答案

按热度按时间
8i9zcol2

8i9zcol21#

  1. version: "2.1"
  2. services:
  3.   jobmanager:
  4.     build:
  5.       context: ./
  6.       args: 
  7.         JAR_FILE: flink-event-tracker-bundled-1.6.0.jar
  8.     image: test/flink-event-tracker
  9.     expose:
  10.       - "6123"
  11.     ports:
  12.       - "8081:8081"
  13.       - "6123:6123"
  14.     command: job-cluster --job-classname com.company.test.flink.pipelines.KafkaPipelineConsumer -Djobmanager.rpc.address=jobmanager --runner=FlinkRunner --streaming=true --checkpointingInterval=30000
  15.     environment:
  16.       - JOB_MANAGER_RPC_ADDRESS=jobmanager
  17.       - JOB_MANAGER=jobmanager
  18.     volumes:
  19.       - data-volume:/docker/volumes
  21.   taskmanager:
  22.     image: test/flink-event-tracker
  23.     expose:
  24.       - "6121"
  25.       - "6122"
  26.     depends_on:
  27.       - jobmanager
  28.     command: task-manager -Djobmanager.rpc.address=jobmanager
  29.     links:
  30.       - "jobmanager:jobmanager"
  31.     environment:
  32.       - JOB_MANAGER_RPC_ADDRESS=jobmanager
  33.       - JOB_MANAGER=jobmanager
  34.     volumes:
  35.       - data-volume:/docker/volumes
  37. volumes:
  38.   data-volume:
  39.     driver: local
  40.     driver_opts:
  41.         o: bind
  42.         type: none
  43.         device: /Users/home/Development/docker/volumes/flink

docker文件

  1. FROM flink:1.9
  3. ARG JAR_FILE=""
  5. ENV APP_OPTS ""
  6. ENV JAVA_OPTS ""
  7. ENV JOB_MANAGER=""
  9. # Build arg allows passing the version at runtime
  11. ARG VERSION=unset-version
  13. COPY flink-conf.yml $FLINK_HOME/conf/flink-conf.yaml
  15. COPY  target/$JAR_FILE $FLINK_HOME/lib/event-tracker.jar
  17. COPY docker-cluster-entrypoint.sh /docker-cluster-entrypoint.sh
  19. RUN apt-get update && apt-get install procps -y && apt-get install curl -y
  20. RUN echo "root:root" | chpasswd
  22. RUN chmod 777 /docker-cluster-entrypoint.sh
  23. RUN chmod 777 $FLINK_HOME/lib/event-tracker.jar
  25. ENTRYPOINT [ "bash","/docker-cluster-entrypoint.sh" ]

docker-cluster-entrypoint.sh文件

  1. FLINK_HOME=${FLINK_HOME:-"/opt/flink/bin"}
  3. JOB_CLUSTER="job-cluster"
  4. TASK_MANAGER="task-manager"
  6. CMD="$1"
  7. shift;
  9. if [ "${CMD}" = "--help" -o "${CMD}" = "-h" ]; then
  10.     echo "Usage: $(basename $0) (${JOB_CLUSTER}|${TASK_MANAGER})"
  11.     exit 0
  12. elif [ "${CMD}" = "${JOB_CLUSTER}" -o "${CMD}" = "${TASK_MANAGER}" ]; then
  13.     echo "Starting the ${CMD}"
  15.     if [ "${CMD}" = "${TASK_MANAGER}" ]; then
  16.         exec $FLINK_HOME/bin/taskmanager.sh start-foreground "$@"
  17.     else
  18.         exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@"
  19.     fi
  20. fi

如何run:-

  1. mvn clean install
  3. docker-compose -f docker-compose.local.yml up --scale taskmanager=2 >  exceptionlog.log
  5. docker-compose -f docker-compose.local.yml build

这是运行docker的整个conf。但是如果您想在kube中运行，只需将docker compose文件转换为相应的kube文件…其余文件可以保持不变。。可能是这样做一个 Helm 库比维修更好。
note:- we 正在使用apachebeam编写作业代码

展开查看全部
赞(0）分享  回复(0）举报  2021-06-24
vfwfrxfs

vfwfrxfs2#

这里有一个完整的工作示例，说明如何在中的kubernetes上创建和运行flink作业集群https://github.com/alpinegizmo/flink-containers-example. 也许这会有帮助。另请参见https://www.youtube.com/watch?v=ceztudgh2te.

赞(0）分享  回复(0）举报  2021-06-24
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com