Hive用结构数组爆炸

eni9jsuy  于 2021-06-26  发布在  Hive
关注(0)|答案(2)|浏览(355)

我正在研究如何在Hive中爆炸一个复杂的类型。我有下面的avro文件,我想用于我的测试,并在上面构建了一个配置单元外部表。
这是我的测试数据。

{"order_id":123456,"customer_id":987654,"total":305,"order_details":[{"quantity":5,"total":55,"product_detail":{"product_id":1000,"product_name":"Hugo Boss XY","product_description": {"string": "Hugo Xy Men 100 ml"}, "product_status": "AVAILABLE", "product_category":["fragrance","perfume"],"price":10.35,"product_hash":"XY123"}},{"quantity":5,"total":250,"product_detail":{"product_id":2000,"product_name":"Cherokee Polo T Shirt","product_description": {"string": "Cherokee Medium Blue Polo T Shirt"}, "product_status": "AVAILABLE", "product_category":["T-shirts","V-Neck","Cotton", "Medium"],"price":50.00,"product_hash":"XY789"}}]}
{"order_id":789012,"customer_id":4567324,"total":220,"order_details":[{"quantity":10,"total":120,"product_detail":{"product_id":1001,"product_name":"Hugo Men Red","product_description": {"string": "Hugo Men Red 150 ml"}, "product_status": "ONLY_FEW_LEFT", "product_category":["fragrance","perfume"],"price":12.99,"product_hash":"XY456"}},{"quantity":10,"total":100,"product_detail":{"product_id":2001,"product_name":"Ruggers Smart","product_description": {"string": "Ruggers Smart White Small Polo T Shirt"}, "product_status": "ONLY_FEW_LEFT", "product_category":["T-shirts","Round-Neck","Woolen", "Small"],"price":9.99,"product_hash":"XY987"}}]}

avro模式

{
   "namespace":"com.treselle.db.model",
   "type":"record",
   "doc":"This Schema describes about Order",
   "name":"Order",
   "fields":[
     {"name":"order_id","type": "long"},
     {"name":"customer_id","type": "long"},
     {"name":"total","type": "float"},
     {"name":"order_details","type":{
        "type":"array",
        "items": {
          "namespace":"com.treselle.db.model",
          "name":"OrderDetail",
          "type":"record",
          "fields": [
            {"name":"quantity","type": "int"},
            {"name":"total","type": "float"},
            {"name":"product_detail","type":{
               "namespace":"com.treselle.db.model",
               "type":"record",
               "name":"Product",
               "fields":[
                 {"name":"product_id","type": "long"},
                 {"name":"product_name","type": "string","doc":"This is the name of the product"},
                 {"name":"product_description","type": ["string", "null"], "default": ""},
                 {"name":"product_status","type": {"name":"product_status", "type": "enum", "symbols": ["AVAILABLE", "OUT_OF_STOCK", "ONLY_FEW_LEFT"]}, "default":"AVAILABLE"},
                 {"name":"product_category","type":{"type": "array", "items": "string"}, "doc": "This contains array of categories"},
                 {"name":"price","type": "float"},
                 {"name": "product_hash", "type": {"type": "fixed", "name": "product_hash", "size": 5}}
               ]
             }
            }
          ]
        }
      }
    }
  ]
}

我的Hive

CREATE EXTERNAL TABLE orders (
    order_id bigint,
    customer_id bigint,
    total float,
    order_items array<
       struct<
          quantity:int,
          total:float,
          product_detail:struct<
             product_id:bigint,
             product_name:string,
             product_description:string,
             product_status:string,
             product_caretogy:array<string>,
             price:float,
             product_hash:binary
          >
       >
    >
)
STORED AS AVRO
LOCATION '/user/hive/test/orders';

查询

SELECT order_id, customer_id FROM orders;

这可以正常工作,并按预期返回两行的结果。
但当我尝试使用横向视图爆炸时,我遇到了问题。

SELECT
   order_id,
   customer_id,
   ord_dets.quantity as line_qty,
   ord_dets.total as line_total
FROM 
   orders 
   LATERAL VIEW explode(order_items) exploded_table as ord_dets;

此查询运行正常,但不会产生任何结果。
这里有什么问题吗?

pod7payv

pod7payv1#

原因是您在模式中定义了 order_items 但是在数据和avro模式中,这个字段被称为 order_details . Hive寻找 order_items 并认为它是一个不存在的字段,默认为null。

bq3bfh9z

bq3bfh9z2#

谢谢你的指点。
当我更正错误时,我在查询时出错了。。。确定失败,出现异常java.io.ioexception:org.apache.avro.avrotypeexception:找到com.treselle.db.model.order\u详细信息,应为union
在进一步分析之后,我发现avro文件中的enum类型和fixed类型都导致了“expecting union”错误。删除这些列之后,我能够成功地查询配置单元表。

相关问题