cloudera5.4.2:使用flume和twitter流时,avro块大小无效或太大

vdzxcuhz  于 2021-06-04  发布在  Flume
关注(0)|答案(1)|浏览(397)

当我尝试cloudera5.4.2时,有一个小问题。基于本文
apacheflume-获取twitter数据http://www.tutorialspoint.com/apache_flume/fetching_twitter_data.htm
它尝试使用flume和twitter流来获取tweet,以便进行数据分析。一切都很好,创建twitter应用程序,在hdfs上创建目录,配置flume,然后开始获取数据,在tweets上创建模式。
那么,问题来了。twitter流媒体将tweets转换为avro格式,并将avro事件发送到downsteamhdfs接收器,当avro支持的hive表加载数据时,我收到错误消息“avro块大小无效或太大”。
哦,什么是avro块和块大小的限制?我能换一下吗?根据这个信息是什么意思?是文件的错吗?是唱片公司的错吗?如果twitter的流媒体遇到了错误数据,它应该会关闭。如果把tweet转换成avro格式是好的,反过来,avro数据应该被正确读取,对吗?
我还尝试了avro-tools-1.7.7.jar

java -jar avro-tools-1.7.7.jar tojson FlumeData.1458090051232

{"id":"710300089206611968","user_friends_count":{"int":1527},"user_location":{"string":"1633"},"user_description":{"string":"Steady Building an Empire..... #UGA"},"user_statuses_count":{"int":44471},"user_followers_count":{"int":2170},"user_name":{"string":"Esquire Shakur"},"user_screen_name":{"string":"Esquire_Bowtie"},"created_at":{"string":"2016-03-16T23:01:52Z"},"text":{"string":"RT @ugaunion: .@ugasga is hosting a debate between the three SGA executive tickets. Learn more about their plans to serve you https://t.co/…"},"retweet_count":{"long":0},"retweeted":{"boolean":true},"in_reply_to_user_id":{"long":-1},"source":{"string":"<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>"},"in_reply_to_status_id":{"long":-1},"media_url_https":null,"expanded_url":null}

{"id":"710300089198088196","user_friends_count":{"int":100},"user_location":{"string":"DM開放してます(`・ω・´)"},"user_description":{"string":"Exception in thread "main" org.apache.avro.AvroRuntimeException: java.io.IOException: Block size invalid or too large for this implementation: -40

at org.apache.avro.file.DataFileStream.hasNextBlock(DataFileStream.java:275)

at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:197)
at org.apache.avro.tool.DataFileReadTool.run(DataFileReadTool.java:77)
at org.apache.avro.tool.Main.run(Main.java:84)
at org.apache.avro.tool.Main.main(Main.java:73)
Caused by: java.io.IOException: Block size invalid or too large for this implementation: -40
at org.apache.avro.file.DataFileStream.hasNextBlock(DataFileStream.java:266)
... 4 more

同样的问题。我用谷歌搜索了很多,根本没有答案。
如果你也遇到过这个问题,谁能给我一个解决办法?或者有人帮你提供一个线索,如果你完全了解avro的东西或推特流下面。
这真是个棘手的问题。好好想想。

rvpgvaaj

rvpgvaaj1#

使用cloudera twittersource
否则就会遇到这个问题。
无法将twitter avro数据正确加载到配置单元表中
本文简介:这是apachetwittersource

TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
Twitter 1% Firehose Source
This source is highly experimental. It connects to the 1% sample Twitter Firehose using streaming API and continuously downloads tweets, converts them to Avro format, and sends Avro events to a downstream Flume sink.

但应该是cloudera twittersource:
https://blog.cloudera.com/blog/2012/09/analyzing-twitter-data-with-hadoop/
http://blog.cloudera.com/blog/2012/10/analyzing-twitter-data-with-hadoop-part-2-gathering-data-with-flume/
http://blog.cloudera.com/blog/2012/11/analyzing-twitter-data-with-hadoop-part-3-querying-semi-structured-data-with-hive/

TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource

而且不只是下载预构建jar,因为我们的cloudera版本是5.4.2,否则会出现以下错误:
由于jar冲突,无法运行flume
您应该使用maven编译它
https://github.com/cloudera/cdh-twitter-example
下载并编译:flume-sources.1.0-snapshot.jar。这个jar包含cloudera twittersource的实现。
步骤:
wget公司https://github.com/cloudera/cdh-twitter-example/archive/master.zip
sudo yum install apache maven put to flume plugins目录:

/var/lib/flume-ng/plugins.d/twitter-streaming/lib/flume-sources-1.0-SNAPSHOT.jar

打包
注意:yum更新到最新版本,否则编译(mvn包)会因为一些安全问题而失败。

相关问题