我一直在尝试使用 Flume 非常适合hadoop集群。另一个选择是 fluentd 但老实说，我不想改变组件以及flume是完美的hdfs和hadoop。
所以，我用 flume-twitter-source-1.6.0.jar 但在开始收集数据的那一刻， flume 无法建立连接。我知道这是cloudera和twitter的一个常见问题，但只要我一直在尝试，就一直找不到答案。不管怎样，我希望有人能帮助理解这个问题。创建自定义源代码是另一种可能的选择，但我并没有真正考虑这一点。

## Defining Components

  TwitterAgent.sources = Twitter
  TwitterAgent.channels = FileChannel
  TwitterAgent.sinks = HDFS 

  ## Component Configuration
  TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
  TwitterAgent.sources.Twitter.channels = FileChannel
  TwitterAgent.sources.Twitter.consumerKey = xxxx
  TwitterAgent.sources.Twitter.consumerSecret = xxxx
  TwitterAgent.sources.Twitter.accessToken = xxxxx
  TwitterAgent.sources.Twitter.accessTokenSecret = xxxx
  TwitterAgent.sources.Twitter.maxFileSize = 10737418240
  TwitterAgent.sources.Twitter.maxBatchSize = 50000
  TwitterAgent.sources.Twitter.maxBatchDurationMillis = 100000
  TwitterAgent.sources.Twitter.keywords = Coronavirus

  TwitterAgent.channels.FileChannel.type = file
  TwitterAgent.channels.FileChannel.checkpointDir = /usr/local/flume/sample_tweets
  TwitterAgent.channels.FileChannel.dataDirs = /usr/local/flume/data
  TwitterAgent.channels.FileChannel.trackerDir = /usr/local/flume/data

  TwitterAgent.sinks.HDFS.channel = FileChannel
  TwitterAgent.sinks.HDFS.type = hdfs
  TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:54310/datasets/covid_tweets
  TwitterAgent.sinks.HDFS.hdfs.filePrefix = tweets-
  TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
  TwitterAgent.sinks.HDFS.hdfs.batchSize = 2000000
  TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
  TwitterAgent.sinks.HDFS.hdfs.rollCount = 20000000

2020-05-31 20:57:14,244 (Twitter Stream consumer-1[Establishing connection]) [DEBUG - twitter4j.internal.logging.SLF4JLogger.debug(SLF4JLogger.java:67)] X-Twitter-Client-URL: http://twitter4j.org/en/twitter4j-3.0.3.xml
2020-05-31 20:57:14,244 (Twitter Stream consumer-1[Establishing connection]) [DEBUG - twitter4j.internal.logging.SLF4JLogger.debug(SLF4JLogger.java:67)] X-Twitter-Client: Twitter4J
2020-05-31 20:57:14,245 (Twitter Stream consumer-1[Establishing connection]) [DEBUG - twitter4j.internal.logging.SLF4JLogger.debug(SLF4JLogger.java:67)] Accept-Encoding: gzip
2020-05-31 20:57:14,245 (Twitter Stream consumer-1[Establishing connection]) [DEBUG - twitter4j.internal.logging.SLF4JLogger.debug(SLF4JLogger.java:67)] User-Agent: twitter4j http://twitter4j.org/ /3.0.3
2020-05-31 20:57:14,245 (Twitter Stream consumer-1[Establishing connection]) [DEBUG - twitter4j.internal.logging.SLF4JLogger.debug(SLF4JLogger.java:67)] X-Twitter-Client-Version: 3.0.3
2020-05-31 20:57:14,245 (Twitter Stream consumer-1[Establishing connection]) [DEBUG - twitter4j.internal.logging.SLF4JLogger.debug(SLF4JLogger.java:67)] Connection: close
2020-05-31 20:57:14,464 (Twitter Stream consumer-1[Establishing connection]) [INFO - twitter4j.internal.logging.SLF4JLogger.info(SLF4JLogger.java:83)] Connection reset
2020-05-31 20:57:14,464 (Twitter Stream consumer-1[Establishing connection]) [ERROR - org.apache.flume.source.twitter.TwitterSource.onException(TwitterSource.java:331)] Exception while streaming tweets
Connection reset
Relevant discussions can be found on the Internet at:
    http://www.google.co.jp/search?q=d0031b0b or
    http://www.google.co.jp/search?q=1db75522
TwitterException{exceptionCode=[d0031b0b-1db75522 db667dea-99334ae4], statusCode=-1, message=null, code=-1, retryAfter=-1, rateLimitStatus=null, version=3.0.3}
    at twitter4j.internal.http.HttpClientImpl.request(HttpClientImpl.java:192)
    at twitter4j.internal.http.HttpClientWrapper.request(HttpClientWrapper.java:61)
    at twitter4j.internal.http.HttpClientWrapper.get(HttpClientWrapper.java:89)
    at twitter4j.TwitterStreamImpl.getSampleStream(TwitterStreamImpl.java:176)
    at twitter4j.TwitterStreamImpl$4.getStream(TwitterStreamImpl.java:164)
    at twitter4j.TwitterStreamImpl$TwitterStreamConsumer.run(TwitterStreamImpl.java:462)
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:196)
    at java.net.SocketInputStream.read(SocketInputStream.java:122)
    at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
    at sun.security.ssl.InputRecord.read(InputRecord.java:480)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:934)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1332)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1359)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1343)
    at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:563)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1301)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
    at twitter4j.internal.http.HttpResponseImpl.<init>(HttpResponseImpl.java:34)
    at twitter4j.internal.http.HttpClientImpl.request(HttpClientImpl.java:156)
    ... 5 more