使用ec2示例iam凭据将flume下沉到s3

ffscu2ro  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(394)

我有一个FlumeFlume可以将数据写入aws s3桶。Flume配置如下

aggregator.sinks.s3LogSink.type=hdfs
   aggregator.sinks.s3LogSink.channel=flumeLogAgentFileChannel
   aggregator.sinks.s3LogSink.hdfs.path=s3n://<aws bucket name>
   aggregator.sinks.s3LogSink.hdfs.fileType=DataStream
   aggregator.sinks.s3LogSink.hdfs.writeFormat=Text
   aggregator.sinks.s3LogSink.hdfs.rollCount = 0
   aggregator.sinks.s3LogSink.hdfs.rollSize = 67108864
   aggregator.sinks.s3LogSink.hdfs.batchSize = 1000
   aggregator.sinks.s3LogSink.hdfs.rollInterval = 0

为了支持Flumes3Flume,我还包括以下jar

commons-codec-1.4.jar
 commons-httpclient-3.1.jar
 hadoop-common-2.7.1.jar
 commons-configuration-1.6.jar
 hadoop-auth-2.7.1.jar
 aws-java-sdk-1.10.64.jar
 aws-java-sdk-s3-1.10.52.jar
 hadoop-aws.jar
 emr-core-2.4.0.jar

当我开始我的Flume过程。。我得到下面的错误

2016-04-18 18:03:06,326 (hdfs-s3LogSink-call-runner-0) [WARN - org.apache.hadoop.util.NativeCodeLoader.<clinit>(NativeCodeLoader.java:62)] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-04-18 18:03:06,359 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:459)] process failed
java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.fs.s3a.S3AFileSystem could not be instantiated
        at java.util.ServiceLoader.fail(ServiceLoader.java:224)
        at java.util.ServiceLoader.access$100(ServiceLoader.java:181)
        at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377)
        at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
        at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2623)
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2634)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
        at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:243)
        at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235)
        at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679)
        at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
        at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError: com/amazonaws/event/ProgressListener
        at java.lang.Class.getDeclaredConstructors0(Native Method)
        at java.lang.Class.privateGetDeclaredConstructors(Class.java:2595)
        at java.lang.Class.getConstructor0(Class.java:2895)
        at java.lang.Class.newInstance(Class.java:354)
        at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373)
        ... 18 more
Caused by: java.lang.ClassNotFoundException: com.amazonaws.event.ProgressListener
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 23 more

有人能帮忙吗?

7ajki6be

7ajki6be1#

铲斗的正确路径如下所示 s3n://<access-key>:<secret-access-key>@<bucket-name>

相关问题