我正在尝试在spark上安装google云存储。我在hadoop类路径中添加了“gcs-connector-latest-hadoop2.jar”，使用 -libjars . GoogleCloudStorageFileSystem 找不到类错误。
$ hadoop fs -libjars /PATH/gcs-connector-hadoop2-latest.jar -ls /HDFS_PATH
20/02/05 05:41:33 WARN fs.FileSystem: Cannot load filesystem: java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem could not be instantiated
20/02/05 05:41:33 WARN fs.FileSystem: java.lang.NoClassDefFoundError: com/google/cloud/hadoop/gcsio/GoogleCloudStorageFileSystem
20/02/05 05:41:33 WARN fs.FileSystem: java.lang.ClassNotFoundException: com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem
20/02/05 05:41:33 WARN fs.FileSystem: Cannot load filesystem: java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem could not be instantiated
20/02/05 05:41:33 WARN fs.FileSystem: java.lang.NoClassDefFoundError: com/google/cloud/hadoop/gcsio/GoogleCloudStorageFileSystem
20/02/05 05:41:33 WARN fs.FileSystem: java.lang.ClassNotFoundException: com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem我有什么遗漏吗？
1条答案
按热度按时间oyxsuwqo1#
如果您对dataproc集群的主节点具有root访问权限，则可以在此处添加gcs连接器和符号链接
/usr/lib/hadoop/lib