我想开始关于Yarn的flink 1.11.2课程。我已经设置了环境变量 HADOOP_CLASSPATH
, HADOOP_CONF_DIR
但是当我发射的时候 ./bin/yarn-session.sh
我有一个错误:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/someuser/flink-1.11.2/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2020-11-26 15:56:35,249 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: fs.hdfs.hadoopconf, /etc/hadoop/conf
2020-11-26 15:56:35,253 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: env.java.home key, /usr/java/latest
2020-11-26 15:56:35,253 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.address, localhost
2020-11-26 15:56:35,253 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.port, 6123
2020-11-26 15:56:35,253 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.memory.process.size, 1600m
2020-11-26 15:56:35,253 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.process.size, 1728m
2020-11-26 15:56:35,253 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2020-11-26 15:56:35,254 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: parallelism.default, 1
2020-11-26 15:56:35,254 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: state.checkpoints.dir, hdfs://some/user/path/.flink11Checkpoint
2020-11-26 15:56:35,254 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.execution.failover-strategy, region
2020-11-26 15:56:35,254 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: rest.port, 8081
2020-11-26 15:56:35,255 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: security.kerberos.login.use-ticket-cache, true
2020-11-26 15:56:35,255 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: security.kerberos.login.keytab, /home/someuser/some.keytab
2020-11-26 15:56:35,255 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: security.kerberos.login.principal, someuser@cluster.com
2020-11-26 15:56:35,255 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: security.kerberos.login.contexts, Client,KafkaClient
2020-11-26 15:56:35,255 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: yarn.tags, streaming
2020-11-26 15:56:35,256 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: env.java.opts, "-server"
2020-11-26 15:56:35,314 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Found Yarn properties file under /tmp/.yarn-properties-someuser.
2020-11-26 15:56:35,812 INFO org.apache.hadoop.security.UserGroupInformation [] - Login successful for user localisation@NETXLR8.FR using keytab file /home/someuser/some.keytab
2020-11-26 15:56:35,812 INFO org.apache.flink.runtime.security.modules.HadoopModule [] - Hadoop user set to someuser@cluster.com (auth:KERBEROS)
2020-11-26 15:56:35,813 INFO org.apache.flink.runtime.security.modules.HadoopModule [] - Kerberos security is enabled and credentials are valid.
2020-11-26 15:56:35,820 INFO org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file will be created as /tmp/jaas-7703404462212921557.conf.
2020-11-26 15:56:35,855 WARN org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The configuration directory ('/home/someuser/flink-1.11.2/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
2020-11-26 15:56:36,332 WARN org.apache.hadoop.security.UserGroupInformation [] - PriviledgedActionException as:someuser@cluster.com (auth:KERBEROS) cause:org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy Yarn session cluster
2020-11-26 15:56:36,333 ERROR org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Error while running the Flink session.
org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy Yarn session cluster
at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:382) ~[flink-dist_2.11-1.11.2.jar:1.11.2]
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:514) ~[flink-dist_2.11-1.11.2.jar:1.11.2]
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$4(FlinkYarnSessionCli.java:751) ~[flink-dist_2.11-1.11.2.jar:1.11.2]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_131]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_131]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924) ~[hadoop-common-2.6.0-cdh5.15.1.jar:?]
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) ~[flink-dist_2.11-1.11.2.jar:1.11.2]
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:751) [flink-dist_2.11-1.11.2.jar:1.11.2]
Caused by: java.lang.ArithmeticException: / by zero
at org.apache.flink.yarn.YarnClusterDescriptor.logIfComponentMemNotIntegerMultipleOfYarnMinAllocation(YarnClusterDescriptor.java:604) ~[flink-dist_2.11-1.11.2.jar:1.11.2]
at org.apache.flink.yarn.YarnClusterDescriptor.validateClusterResources(YarnClusterDescriptor.java:559) ~[flink-dist_2.11-1.11.2.jar:1.11.2]
at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:506) ~[flink-dist_2.11-1.11.2.jar:1.11.2]
at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:375) ~[flink-dist_2.11-1.11.2.jar:1.11.2]
... 7 more
------------------------------------------------------------
The program finished with the following exception:
org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy Yarn session cluster
at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:382)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:514)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$4(FlinkYarnSessionCli.java:751)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:751)
Caused by: java.lang.ArithmeticException: / by zero
at org.apache.flink.yarn.YarnClusterDescriptor.logIfComponentMemNotIntegerMultipleOfYarnMinAllocation(YarnClusterDescriptor.java:604)
at org.apache.flink.yarn.YarnClusterDescriptor.validateClusterResources(YarnClusterDescriptor.java:559)
at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:506)
at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:375)
... 7 more
hadoop版本:hadoop 2.6.0-cdh5.15.1
我们已经在同一个集群上运行了Flink1.7.0。
我应该改变什么才能正常运行?
是否可以在同一个Yarn簇上运行两个版本的flink?
更新:
我可以开始Flink的故事 1.9.3
以及 1.10.2
在同一个集群上但不是 1.11.2
我还是不知道该换什么来运行它。
暂无答案!
目前还没有任何答案,快来回答吧!