我们使用hiveserver2(在hortonworkshdp2.2发行版上)创建新的claster。一段时间后,我们在hdfs上的/tmp/hive/hive中有超过1048576个目录,因为hive服务器在这个位置生成它。
有人有类似的问题吗?来自hiveserver的日志:
2015-08-31 06:48:15,828 WARN [HiveServer2-Handler-Pool: Thread-1104]: conf.HiveConf (HiveConf.java:initialize(2499)) - HiveConf of name hive.heapsize does not exist
2015-08-31 06:48:15,829 WARN [HiveServer2-Handler-Pool: Thread-1104]: conf.HiveConf (HiveConf.java:initialize(2499)) - HiveConf of name hive.server2.enable.impersonation does not exist
2015-08-31 06:48:15,829 WARN [HiveServer2-Handler-Pool: Thread-1104]: conf.HiveConf (HiveConf.java:initialize(2499)) - HiveConf of name hive.auto.convert.sortmerge.join.noconditionaltask does not exist
2015-08-31 06:48:15,833 INFO [HiveServer2-Handler-Pool: Thread-1104]: thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(232)) - Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V6
2015-08-31 06:48:15,835 INFO [HiveServer2-Handler-Pool: Thread-1104]: session.SessionState (SessionState.java:createPath(558)) - Created local directory: /tmp/ffd9e5e7-7a4e-472e-b5f1-9c7f8acb0bff_resources
2015-08-31 06:48:15,883 INFO [HiveServer2-Handler-Pool: Thread-1104]: session.SessionState (SessionState.java:createPath(558)) - Created HDFS directory: /tmp/hive/hive/ffd9e5e7-7a4e-472e-b5f1-9c7f8acb0bff
2015-08-31 06:48:15,884 INFO [HiveServer2-Handler-Pool: Thread-1104]: session.SessionState (SessionState.java:createPath(558)) - Created local directory: /tmp/hive/ffd9e5e7-7a4e-472e-b5f1-9c7f8acb0bff
2015-08-31 06:48:16,064 INFO [HiveServer2-Handler-Pool: Thread-1104]: session.SessionState (SessionState.java:createPath(558)) - Created HDFS directory: /tmp/hive/hive/ffd9e5e7-7a4e-472e-b5f1-9c7f8acb0bff/_tmp_space.db
2015-08-31 06:48:16,065 INFO [HiveServer2-Handler-Pool: Thread-1104]: session.SessionState (SessionState.java:start(460)) - No Tez session required at this point. hive.execution.engine=mr.
创建会话时的hiveserver方法:
/**
* Create dirs & session paths for this session:
* 1. HDFS scratch dir
* 2. Local scratch dir
* 3. Local downloaded resource dir
* 4. HDFS session path
* 5. Local session path
* 6. HDFS temp table space
* @param userName
* @throws IOException
*/
private void createSessionDirs(String userName) throws IOException {
HiveConf conf = getConf();
Path rootHDFSDirPath = createRootHDFSDir(conf);
// Now create session specific dirs
String scratchDirPermission = HiveConf.getVar(conf, HiveConf.ConfVars.SCRATCHDIRPERMISSION);
Path path;
// 1. HDFS scratch dir
path = new Path(rootHDFSDirPath, userName);
hdfsScratchDirURIString = path.toUri().toString();
createPath(conf, path, scratchDirPermission, false, false);
// 2. Local scratch dir
path = new Path(HiveConf.getVar(conf, HiveConf.ConfVars.LOCALSCRATCHDIR));
createPath(conf, path, scratchDirPermission, true, false);
// 3. Download resources dir
path = new Path(HiveConf.getVar(conf, HiveConf.ConfVars.DOWNLOADED_RESOURCES_DIR));
createPath(conf, path, scratchDirPermission, true, false);
// Finally, create session paths for this session
// Local & non-local tmp location is configurable. however it is the same across
// all external file systems
String sessionId = getSessionId();
// 4. HDFS session path
hdfsSessionPath = new Path(hdfsScratchDirURIString, sessionId);
createPath(conf, hdfsSessionPath, scratchDirPermission, false, true);
conf.set(HDFS_SESSION_PATH_KEY, hdfsSessionPath.toUri().toString());
// 5. Local session path
localSessionPath = new Path(HiveConf.getVar(conf, HiveConf.ConfVars.LOCALSCRATCHDIR), sessionId);
createPath(conf, localSessionPath, scratchDirPermission, true, true);
conf.set(LOCAL_SESSION_PATH_KEY, localSessionPath.toUri().toString());
// 6. HDFS temp table space
hdfsTmpTableSpace = new Path(hdfsSessionPath, TMP_PREFIX);
createPath(conf, hdfsTmpTableSpace, scratchDirPermission, false, true);
conf.set(TMP_TABLE_SPACE_KEY, hdfsTmpTableSpace.toUri().toString());
}
2条答案
按热度按时间s71maibg1#
这里有答案。https://issues.apache.org/jira/browse/hive-15068
更高版本的Hive解决了这个问题。对于较低版本,那么,编写cron作业就可以了
vi4fp9gy2#
我们之前也面临类似的问题。许多配置单元在运行配置单元客户端和默认hdfs示例的计算机上都使用临时文件夹。这些文件夹用于存储每个查询的临时/中间数据集,通常在查询完成时由配置单元客户端进行清理。但是,在异常终止配置单元客户端的情况下,可能会留下一些数据。具体配置如下:
在hdfs集群上,这被默认设置为/tmp/hive,并由客户机上的配置变量hive.exec.scratchdir控制,它被硬编码为/tmp/注意,在将数据写入表/分区时,hive将首先写入目标表文件系统上的临时位置(使用hive.exec.scratchdir作为临时位置),然后将数据移动到目标表。这适用于所有情况—无论表是存储在hdfs(正常情况)中,还是存储在文件系统(如s3甚至nfs)中。
来源
因此,您可以使用手动脚本或作业定期清理temp位置,也可以cron shell脚本清理30或60天的数据