flink 1.10未连接到minio(s3)

kyks70gy  于 2021-06-21  发布在  Flink
关注(0)|答案(0)|浏览(821)

我正在本地运行docker,运行flink和minio当我尝试连接到minio时,总是出现以下错误:
原因:org.apache.flink.core.fs.unsupportedfilesystemschemeexception:找不到方案“s3”的文件系统实现。flink不直接支持该方案,也无法加载支持该方案的hadoop文件系统。
似乎插件加载不正确。
我的flink配置(flink-conf.yaml):

state.backend: filesystem
s3.endpoint: http://minio:9000
s3.path.style.access: true
s3.access-key: minio
s3.secret-key: minio123

presto.s3.access-key: minio 
presto.s3.secret-key: minio123  
presto.s3.endpoint: http://minio:9000   
presto.s3.path-style-access: true

我复制了所需的插件,如下所示:

mkdir -p plugins/s3-fs-presto
cp opt/flink-s3-fs-presto-*.jar plugins/s3-fs-presto

有什么建议吗?
堆栈跟踪:
程序已完成,但出现以下异常:

org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: 7ae6657256719d8c32d76ba113fb35f0)
    at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:262)
    at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:338)
    at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:326)
    at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62)
    at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:820)
    at org.apache.flink.api.java.DataSet.collect(DataSet.java:413)
    at org.apache.flink.api.java.DataSet.print(DataSet.java:1652)
    at org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:96)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:604)
    at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:466)
    at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274)
    at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746)
    at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273)
    at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205)
    at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1008)
    at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1081)
    at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
    at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1081)
Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
    at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:147)
    at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:259)
    ... 21 more
Caused by: java.io.IOException: Error opening the Input Split s3://test/test.txt [0,3243]: Could not find a file system implementation for scheme 's3'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.
    at org.apache.flink.api.common.io.FileInputFormat.open(FileInputFormat.java:824)
    at org.apache.flink.api.common.io.DelimitedInputFormat.open(DelimitedInputFormat.java:470)
    at org.apache.flink.api.common.io.DelimitedInputFormat.open(DelimitedInputFormat.java:47)
    at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:173)
    at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705)
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 's3'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.
    at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:450)
    at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:362)
    at org.apache.flink.api.common.io.FileInputFormat$InputSplitOpenThread.run(FileInputFormat.java:995)
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies.
    at org.apache.flink.core.fs.UnsupportedSchemeFactory.create(UnsupportedSchemeFactory.java:58)
    at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:446)
    ... 2 more

# 

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题