spark不能处理递归avro数据

vybvopom  于 2021-05-17  发布在  Spark
关注(0)|答案(1)|浏览(538)

我有 avsc 架构如下:

{
  "name": "address",
  "type": [
     "null", 
        {
           "type":"record",
           "name":"Address",
           "namespace":"com.data",
           "fields":[
              {
                 "name":"address",
                 "type":[ "null","com.data.Address"],
                 "default":null
              }
           ]
        }
     ],
  "default": null
}

在pyspark中加载此数据时:

jsonFormatSchema = open("Address.avsc", "r").read()

spark = SparkSession.builder.appName('abc').getOrCreate()
df = spark.read.format("avro")\
.option("avroSchema", jsonFormatSchema)\
.load("xxx.avro")

我有个例外:

"Found recursive reference in Avro schema, which can not be processed by Spark"

我尝试了许多其他配置,但都没有成功。要执行,我使用spark submit:

--packages org.apache.spark:spark-avro_2.12:3.0.1
jtw3ybtb

jtw3ybtb1#

这是一个预期的功能,您可以看看“问题”:
https://issues.apache.org/jira/browse/spark-25718

相关问题