我想收集用户在使用sts查询时使用的表,以便清理不必要的表。我写了这样一个演示:
public class CustomHook implements ExecuteWithHookContext {
private static final Logger LOGGER = LoggerFactory.getLogger(CustomHook.class);
private static final HashSet<String> OPERATION_NAMES = new HashSet<>();
static {
OPERATION_NAMES.add(HiveOperation.CREATETABLE.getOperationName());
OPERATION_NAMES.add(HiveOperation.ALTERDATABASE.getOperationName());
OPERATION_NAMES.add(HiveOperation.ALTERDATABASE_OWNER.getOperationName());
OPERATION_NAMES.add(HiveOperation.ALTERTABLE_ADDCOLS.getOperationName());
OPERATION_NAMES.add(HiveOperation.ALTERTABLE_LOCATION.getOperationName());
OPERATION_NAMES.add(HiveOperation.ALTERTABLE_PROPERTIES.getOperationName());
OPERATION_NAMES.add(HiveOperation.ALTERTABLE_RENAME.getOperationName());
OPERATION_NAMES.add(HiveOperation.ALTERTABLE_RENAMECOL.getOperationName());
OPERATION_NAMES.add(HiveOperation.ALTERTABLE_REPLACECOLS.getOperationName());
OPERATION_NAMES.add(HiveOperation.CREATEDATABASE.getOperationName());
OPERATION_NAMES.add(HiveOperation.DROPDATABASE.getOperationName());
OPERATION_NAMES.add(HiveOperation.DROPTABLE.getOperationName());
}
@Override
public void run(HookContext hookContext) throws Exception {
assert (hookContext.getHookType() == HookType.POST_EXEC_HOOK);
QueryPlan plan = hookContext.getQueryPlan();
String operationName = plan.getOperationName();
logWithHeader("Query executed: " + plan.getQueryString());
logWithHeader("Operation: " + operationName);
if (OPERATION_NAMES.contains(operationName)
&& !plan.isExplain()) {
logWithHeader("Monitored Operation");
Set<ReadEntity> inputs = hookContext.getInputs();
Set<WriteEntity> outputs = hookContext.getOutputs();
for (Entity entity : inputs) {
logWithHeader("Hook metadata input value: " + toJson(entity));
}
for (Entity entity : outputs) {
logWithHeader("Hook metadata output value: " + toJson(entity));
}
} else {
logWithHeader("Non-monitored Operation, ignoring hook");
}
}
private static String toJson(Entity entity) throws Exception {
ObjectMapper mapper = new ObjectMapper();
switch (entity.getType()) {
case DATABASE:
Database db = entity.getDatabase();
return mapper.writeValueAsString(db);
case TABLE:
return mapper.writeValueAsString(entity.getTable().getTTable());
}
return null;
}
private void logWithHeader(Object obj){
LOGGER.info("[CustomHook][Thread: "+Thread.currentThread().getName()+"] | " + obj);
}
}
我将它打包并设置在hiveconf的hive-site.xml中,然后将它放在hive/lib/中。当我使用spark thrift server进行查询时,什么都没有发生。我想知道sts是否支持Hive钩?如果不支持,是否有其他方法来收集sts中查询使用的表?
版本:
连接到:spark sql(版本2.4.7-amzn-0)
驱动程序:hive jdbc(版本1.2.1-spark2-amzn-4)
暂无答案!
目前还没有任何答案,快来回答吧!