我在尝试使用flink的StreamingFileLink将文件放入azure datalake gen 2时遇到问题,我正在使用core-site.xml和hadoop批量格式,我正在尝试使用abfss://格式复制到我的datalake(也可以使用abfs://)
java.lang.UnsupportedOperationException: Recoverable writers on Hadoop are only supported for HDFS
[job-playground-job-cluster-0 flink-job-cluster] at org.apache.flink.runtime.fs.hdfs.HadoopRecoverableWriter.<init>(HadoopRecoverableWriter.java:61) ~[flink-dist_2.11-1.11.0.jar:1.11.0]
[job-playground-job-cluster-0 flink-job-cluster] at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.createRecoverableWriter(HadoopFileSystem.java:202) ~[flink-dist_2.11-1.11.0.jar:1.11.0]
[job-playground-job-cluster-0 flink-job-cluster] at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.createRecoverableWriter(SafetyNetWrapperFileSystem.java:69) ~[flink-dist_2.11-1.11.0.jar:1.11.0]
[job-playground-job-cluster-0 flink-job-cluster] at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$BulkFormatBuilder.createBuckets(StreamingFileSink.java:371) ~[flink-dist_2.11-1.11.0.jar:1.11.0]
我阅读了官方文件,潜入图书馆,问题就在这里:https://github.com/apache/flink/blob/master/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/hadooprecoverablewriter.java#l60
public HadoopRecoverableWriter(org.apache.hadoop.fs.FileSystem fs) {
this.fs = checkNotNull(fs);
// This writer is only supported on a subset of file systems
if (!"hdfs".equalsIgnoreCase(fs.getScheme())) {
throw new UnsupportedOperationException(
"Recoverable writers on Hadoop are only supported for HDFS");
}
这是我的core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.azure.account.auth.type.ADLS_ACCOUNT_NAME.dfs.core.windows.net</name>
<value>SharedKey</value>
<description>
It is inferred by the url
</description>
</property>
<property>
<name>fs.azure.account.key.ADLS_ACCOUNT_NAME.dfs.core.windows.net</name>
<value>ADLS_KEY</value>
<description>
</description>
</property>
<property>
<name>fs.azure.createRemoteFileSystemDuringInitialization</name>
<value>true</value>
</property>
<property>
<name>fs.azure.always.use.https</name>
<value>true</value>
</property>
</configuration>
任何人都通过了这个问题,或者是扩展abfss/abfs的问题。
1条答案
按热度按时间hm2xizp91#
StreamingFileLink尚不支持azure数据湖。