Kafka S3提供程序的文件脉冲连接器错误(源连接器)

iugsix8n  于 2023-02-15  发布在  Apache
关注(0)|答案(1)|浏览(136)

我正在尝试使用Filepulse源连接器从S3 bucket轮询csv文件。当任务开始时,我收到以下错误。我需要添加哪些其他库才能从S3 bucket执行此操作?下面是配置文件。
我哪里做错了?

Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:208)
java.nio.file.FileSystemNotFoundException: Provider "s3" not installed
at java.base/java.nio.file.Path.of(Path.java:212)
at java.base/java.nio.file.Paths.get(Paths.java:98)
at io.streamthoughts.kafka.connect.filepulse.fs.reader.LocalFileStorage.exists(LocalFileStorage.java:62)

配置文件:

{
  "name": "FilePulseConnector_3",
  "config": {
  "connector.class": "io.streamthoughts.kafka.connect.filepulse.source.FilePulseSourceConnector",
  "filters": "ParseCSVLine, Drop",
  "filters.Drop.if": "{{ equals($value.artist, 'U2') }}",
  "filters.Drop.invert": "true",
  "filters.Drop.type": "io.streamthoughts.kafka.connect.filepulse.filter.DropFilter",
  "filters.ParseCSVLine.extract.column.name": "headers",
  "filters.ParseCSVLine.trim.column": "true",
  "filters.ParseCSVLine.seperator": ";",
  "filters.ParseCSVLine.type": "io.streamthoughts.kafka.connect.filepulse.filter.DelimitedRowFilter",
  "fs.cleanup.policy.class": "io.streamthoughts.kafka.connect.filepulse.fs.clean.LogCleanupPolicy",
  "fs.cleanup.policy.triggered.on":"COMMITTED",
  "fs.listing.class": "io.streamthoughts.kafka.connect.filepulse.fs.AmazonS3FileSystemListing",
  "fs.listing.filters":"io.streamthoughts.kafka.connect.filepulse.fs.filter.RegexFileListFilter",
  "fs.listing.interval.ms": "10000",
  "file.filter.regex.pattern":".*\\.csv$",
  "offset.policy.class":"io.streamthoughts.kafka.connect.filepulse.offset.DefaultSourceOffsetPolicy",
  "offset.attributes.string": "name",
  "skip.headers": "1",
  "topic": "connect-file-pulse-quickstart-csv",
  "tasks.reader.class": "io.streamthoughts.kafka.connect.filepulse.fs.reader.LocalRowFileInputReader",
  "tasks.file.status.storage.class": "io.streamthoughts.kafka.connect.filepulse.state.KafkaFileObjectStateBackingStore",
  "tasks.file.status.storage.bootstrap.servers": "172.27.157.66:9092",
  "tasks.file.status.storage.topic": "connect-file-pulse-status",
  "tasks.file.status.storage.topic.partitions": 10,
  "tasks.file.status.storage.topic.replication.factor": 1,
  "tasks.max": 1,
  "aws.access.key.id":"<<>>",
  "aws.secret.access.key":"<<>>",
  "aws.s3.bucket.name":"mytestbucketamtrak",
  "aws.s3.region":"us-east-1"
}
}

我应该在库中放入什么才能让它工作?镜头连接器来源于s3桶没有问题。所以这不是一个凭据问题。

3htmauhk

3htmauhk1#

正如@OneCricketeer建议你关注的评论中提到的那样--github.com/streamthoughts/kafka-connect-file-pulse/issues/382指出了根本原因。
将配置文件修改为使用此属性来源文件:
“任务.阅读器.类”:“连接。文件脉冲。文件读取器。亚马逊S3行文件输入读取器”

相关问题