Spark结构化流- Kinesis流

pbpqsu0x  于 2023-08-06  发布在  Apache
关注(0)|答案(1)|浏览(112)

Spark是否支持使用Kinesis流作为数据源的结构化流?Databricks版本似乎支持-https://docs.databricks.com/structured-streaming/kinesis-best-practices.html。但是,Databricks之外的Spark是否支持此功能?

vq8itlhq

vq8itlhq1#

可以,您可以使用以下开源连接器:https://github.com/roncemer/spark-sql-kinesis
范例:

// Stream data from the "test" stream
// Note: if running on AWS EC2, you can omit the secret and access keys in lieu of the attached IAM role on the EC2 instance

val kinesis = spark
    .readStream
    .format("kinesis")
    .option("streamName", "spark-streaming-example")
    .option("endpointUrl", "https://kinesis.us-east-1.amazonaws.com")
    .option("awsAccessKeyId", [ACCESS_KEY])
    .option("awsSecretKey", [SECRET_KEY])
    .option("startingposition", "TRIM_HORIZON")
    .load

字符串

相关问题