flink的版本:1.4.0、1.4.1、1.4.2
当我尝试制作这个简单的flink应用程序时
val env: StreamExecutionEnvironment =
StreamExecutionEnvironment.getExecutionEnvironment
env.fromElements("a", "b", "c").addSink(new BucketingSink[String]("file:///Users/joshlemer/projects/my-project/target/output"))
我得到以下运行时异常:
Exception in thread "main" java.lang.NoClassDefFoundError: Lorg/apache/hadoop/fs/FileSystem;
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
at java.lang.Class.getDeclaredFields(Class.java:1916)
at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:72)
at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.clean(StreamExecutionEnvironment.java:1550)
at org.apache.flink.streaming.api.datastream.DataStream.clean(DataStream.java:184)
at org.apache.flink.streaming.api.datastream.DataStream.addSink(DataStream.java:1134)
at org.apache.flink.streaming.api.scala.DataStream.addSink(DataStream.scala:1036)
at com.company.project.Job$.run(Job.scala:52)
at com.company.project.Job$.main(Job.scala:28)
at com.company.project.Job.main(Job.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FileSystem
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
即使我可以用 dataStream.writeAsText(...)
.
我的build.sbt也很典型:
val flinkVersion = "1.4.2"
val flinkDependencies =
Seq(
"org.apache.flink" %% "flink-scala" % flinkVersion % "provided",
"org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided",
"org.apache.flink" %% "flink-statebackend-rocksdb" % flinkVersion,
"org.apache.flink" %% "flink-connector-kafka-0.11" % flinkVersion,
"org.apache.flink" %% "flink-connector-filesystem" % flinkVersion,
"org.apache.flink" %% "flink-test-utils" % flinkVersion % "test",
"org.apache.flink" % "flink-test-utils-junit" % flinkVersion % "test"
)
额外的 idea.sbt
正如flink为intellij用户推荐的那样
lazy val mainRunner = project.in(file("mainRunner")).dependsOn(RootProject(file("."))).settings(
// we set all provided dependencies to none, so that they are included in the classpath of mainRunner
libraryDependencies := (libraryDependencies in RootProject(file("."))).value.map{
module =>
if (module.configurations.equals(Some("provided"))) {
module.copy(configurations = None)
} else {
module
}
}
)
我用它来运行应用程序(mainrunner被设置为应用程序类路径)。
我很困惑为什么会发生这种情况,尤其是为什么程序包以“lorg”而不是“org”开头?
谢谢!
1条答案
按热度按时间xqkwcwgp1#
在1.4发行说明中:
从版本1.4开始,flink可以在类路径中不存在任何hadoop依赖项的情况下运行。除了在没有hadoop的情况下简单地运行之外,这使flink能够动态地使用类路径中可用的hadoop版本。
例如,您可以下载flink的hadoop免费版本,但可以使用它在任何受支持的yarn版本上运行,flink将动态地使用yarn的hadoop依赖项。
这也意味着,在使用hdfs连接器的情况下,比如bucketingsink或rollingsink,现在必须确保使用带有捆绑hadoop依赖项的flink发行版,或者确保在为应用程序构建jar文件时包含hadoop依赖项。