flume无法将文件放入s3 bucket

vlju58qv  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(443)

我正在使用flume的node-only(测试)模式;flume从rabbitmq中提取消息并将其放入amazons3存储桶。

问题是:

flume实际上从rabbitmq中提取,但文件不会出现在s3 bucket中。

技术细节:

我按以下方式启动Flume:

flume node -1 -c $FQDN':amqp("exchangeName=[exchange name]", "bindings=[binding name]", "host=127.0.0.1", "port=5672", "userName=[user]", "password=[pass]", "exchangeType=direct", "durableExchange=false", "queueName=[queue name]", "durableQueue=true", "exclusiveQueue=false", "autoDeleteQueue=false", "useMessageTimestamp=true")|collectorSink("s3n://[Amazon key]:[Amazon secret]@[path at S3]","server");' -s "$@" > "$log" 2>&1

Flume日志:

flume重启后,通过相关的exchange和队列发送内容时,flume日志中会出现以下行:

INFO com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink: Opening s3n://[key]:[secret]@[path at S3]

WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library not loaded
kiayqfof

kiayqfof1#

你不会相信的。。。
flume无法管理包含“/”的amazon密钥,我使用的密钥包含一个。然后flume意识到了这一点,只是没有给亚马逊打电话。
解决方案:更改amazon密钥,直到得到一个不带“/”的密钥

相关问题