我正在研究如何在Hive中爆炸一个复杂的类型。我有下面的avro文件,我想用于我的测试,并在上面构建了一个配置单元外部表。
这是我的测试数据。
{"order_id":123456,"customer_id":987654,"total":305,"order_details":[{"quantity":5,"total":55,"product_detail":{"product_id":1000,"product_name":"Hugo Boss XY","product_description": {"string": "Hugo Xy Men 100 ml"}, "product_status": "AVAILABLE", "product_category":["fragrance","perfume"],"price":10.35,"product_hash":"XY123"}},{"quantity":5,"total":250,"product_detail":{"product_id":2000,"product_name":"Cherokee Polo T Shirt","product_description": {"string": "Cherokee Medium Blue Polo T Shirt"}, "product_status": "AVAILABLE", "product_category":["T-shirts","V-Neck","Cotton", "Medium"],"price":50.00,"product_hash":"XY789"}}]}
{"order_id":789012,"customer_id":4567324,"total":220,"order_details":[{"quantity":10,"total":120,"product_detail":{"product_id":1001,"product_name":"Hugo Men Red","product_description": {"string": "Hugo Men Red 150 ml"}, "product_status": "ONLY_FEW_LEFT", "product_category":["fragrance","perfume"],"price":12.99,"product_hash":"XY456"}},{"quantity":10,"total":100,"product_detail":{"product_id":2001,"product_name":"Ruggers Smart","product_description": {"string": "Ruggers Smart White Small Polo T Shirt"}, "product_status": "ONLY_FEW_LEFT", "product_category":["T-shirts","Round-Neck","Woolen", "Small"],"price":9.99,"product_hash":"XY987"}}]}
avro模式
{
"namespace":"com.treselle.db.model",
"type":"record",
"doc":"This Schema describes about Order",
"name":"Order",
"fields":[
{"name":"order_id","type": "long"},
{"name":"customer_id","type": "long"},
{"name":"total","type": "float"},
{"name":"order_details","type":{
"type":"array",
"items": {
"namespace":"com.treselle.db.model",
"name":"OrderDetail",
"type":"record",
"fields": [
{"name":"quantity","type": "int"},
{"name":"total","type": "float"},
{"name":"product_detail","type":{
"namespace":"com.treselle.db.model",
"type":"record",
"name":"Product",
"fields":[
{"name":"product_id","type": "long"},
{"name":"product_name","type": "string","doc":"This is the name of the product"},
{"name":"product_description","type": ["string", "null"], "default": ""},
{"name":"product_status","type": {"name":"product_status", "type": "enum", "symbols": ["AVAILABLE", "OUT_OF_STOCK", "ONLY_FEW_LEFT"]}, "default":"AVAILABLE"},
{"name":"product_category","type":{"type": "array", "items": "string"}, "doc": "This contains array of categories"},
{"name":"price","type": "float"},
{"name": "product_hash", "type": {"type": "fixed", "name": "product_hash", "size": 5}}
]
}
}
]
}
}
}
]
}
我的Hive
CREATE EXTERNAL TABLE orders (
order_id bigint,
customer_id bigint,
total float,
order_items array<
struct<
quantity:int,
total:float,
product_detail:struct<
product_id:bigint,
product_name:string,
product_description:string,
product_status:string,
product_caretogy:array<string>,
price:float,
product_hash:binary
>
>
>
)
STORED AS AVRO
LOCATION '/user/hive/test/orders';
查询
SELECT order_id, customer_id FROM orders;
这可以正常工作,并按预期返回两行的结果。
但当我尝试使用横向视图爆炸时,我遇到了问题。
SELECT
order_id,
customer_id,
ord_dets.quantity as line_qty,
ord_dets.total as line_total
FROM
orders
LATERAL VIEW explode(order_items) exploded_table as ord_dets;
此查询运行正常,但不会产生任何结果。
这里有什么问题吗?
2条答案
按热度按时间pod7payv1#
原因是您在模式中定义了
order_items
但是在数据和avro模式中,这个字段被称为order_details
. Hive寻找order_items
并认为它是一个不存在的字段,默认为null。bq3bfh9z2#
谢谢你的指点。
当我更正错误时,我在查询时出错了。。。确定失败,出现异常java.io.ioexception:org.apache.avro.avrotypeexception:找到com.treselle.db.model.order\u详细信息,应为union
在进一步分析之后,我发现avro文件中的enum类型和fixed类型都导致了“expecting union”错误。删除这些列之后,我能够成功地查询配置单元表。