org.apache.hadoop.hive.ql.metadata.Table.getNumBuckets()方法的使用及代码示例

x33g5p2x  于2022-01-29 转载在 其他  
字(13.3k)|赞(0)|评价(0)|浏览(125)

本文整理了Java中org.apache.hadoop.hive.ql.metadata.Table.getNumBuckets()方法的一些代码示例，展示了Table.getNumBuckets()的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。Table.getNumBuckets()方法的具体详情如下：
包路径：org.apache.hadoop.hive.ql.metadata.Table
类名称：Table
方法名：getNumBuckets

Table.getNumBuckets介绍

暂无

代码示例

代码示例来源：origin: apache/incubator-gobblin

  1. public TableLikeStageableTableMetadata(Table referenceTable, String destinationDB, String destinationTableName, String targetDataPath) {
  2.  super(destinationTableName, destinationTableName + "_STAGING", destinationDB, targetDataPath,
  3.    getTableProperties(referenceTable), new ArrayList<>(), Optional.of(referenceTable.getNumBuckets()), new Properties(), false, Optional.absent(),
  4.    new ArrayList<>());
  5. }

代码示例来源：origin: apache/incubator-gobblin

  1. public TableLikeStageableTableMetadata(Table referenceTable, Config config) {
  2.  super(HiveDataset.resolveTemplate(config.getString(StageableTableMetadata.DESTINATION_TABLE_KEY), referenceTable),
  3.    HiveDataset.resolveTemplate(config.getString(StageableTableMetadata.DESTINATION_TABLE_KEY), referenceTable) + "_STAGING",
  4.    HiveDataset.resolveTemplate(config.getString(StageableTableMetadata.DESTINATION_DB_KEY), referenceTable),
  5.    HiveDataset.resolveTemplate(config.getString(DESTINATION_DATA_PATH_KEY), referenceTable),
  6.    getTableProperties(referenceTable), new ArrayList<>(), Optional.of(referenceTable.getNumBuckets()),
  7.    new Properties(), false, Optional.absent(), new ArrayList<>());
  8. }

代码示例来源：origin: apache/hive

  1. @SuppressWarnings("nls")
  2. public FileStatus[] getSortedPaths() {
  3.  try {
  4.   // Previously, this got the filesystem of the Table, which could be
  5.   // different from the filesystem of the partition.
  6.   FileSystem fs = FileSystem.get(getPath().toUri(), SessionState.getSessionConf());
  7.   String pathPattern = getPath().toString();
  8.   if (getNumBuckets() > 0) {
  9.    pathPattern = pathPattern + "/*";
  10.   }
  11.   LOG.info("Path pattern = " + pathPattern);
  12.   FileStatus srcs[] = fs.globStatus(new Path(pathPattern), FileUtils.HIDDEN_FILES_PATH_FILTER);
  13.   Arrays.sort(srcs);
  14.   for (FileStatus src : srcs) {
  15.    LOG.info("Got file: " + src.getPath());
  16.   }
  17.   if (srcs.length == 0) {
  18.    return null;
  19.   }
  20.   return srcs;
  21.  } catch (Exception e) {
  22.   throw new RuntimeException("Cannot get path ", e);
  23.  }
  24. }

代码示例来源：origin: apache/drill

  1. @SuppressWarnings("nls")
  2. public FileStatus[] getSortedPaths() {
  3.  try {
  4.   // Previously, this got the filesystem of the Table, which could be
  5.   // different from the filesystem of the partition.
  6.   FileSystem fs = FileSystem.get(getPath().toUri(), SessionState.getSessionConf());
  7.   String pathPattern = getPath().toString();
  8.   if (getNumBuckets() > 0) {
  9.    pathPattern = pathPattern + "/*";
  10.   }
  11.   LOG.info("Path pattern = " + pathPattern);
  12.   FileStatus srcs[] = fs.globStatus(new Path(pathPattern), FileUtils.HIDDEN_FILES_PATH_FILTER);
  13.   Arrays.sort(srcs);
  14.   for (FileStatus src : srcs) {
  15.    LOG.info("Got file: " + src.getPath());
  16.   }
  17.   if (srcs.length == 0) {
  18.    return null;
  19.   }
  20.   return srcs;
  21.  } catch (Exception e) {
  22.   throw new RuntimeException("Cannot get path ", e);
  23.  }
  24. }

代码示例来源：origin: apache/hive

  1. private FileSinkDesc getFileSinkDesc(Path tempDirPath) {
  2.  Table table = mock(Table.class);
  3.  when(table.getNumBuckets()).thenReturn(NUM_BUCKETS);
  4.  FileSinkDesc conf = new FileSinkDesc(tempDirPath, null, false);
  5.  conf.setTable(table);
  6.  return conf;
  7. }

代码示例来源：origin: apache/hive

  1. /**
  2.  * Remove all temporary files and duplicate (double-committed) files from a given directory.
  3.  *
  4.  * @return a list of path names corresponding to should-be-created empty buckets.
  5.  */
  6. public static List<Path> removeTempOrDuplicateFiles(FileSystem fs, FileStatus[] fileStats,
  7.   DynamicPartitionCtx dpCtx, FileSinkDesc conf, Configuration hconf, Set<Path> filesKept, boolean isBaseDir)
  8.     throws IOException {
  9.  int dpLevels = dpCtx == null ? 0 : dpCtx.getNumDPCols(),
  10.    numBuckets = (conf != null && conf.getTable() != null) ? conf.getTable().getNumBuckets() : 0;
  11.  return removeTempOrDuplicateFiles(
  12.    fs, fileStats, null, dpLevels, numBuckets, hconf, null, 0, false, filesKept, isBaseDir);
  13. }

代码示例来源：origin: apache/hive

  1. public Integer splitCount(HiveTableScan scan, RelMetadataQuery mq) {
  2.  Integer splitCount;
  3.  RelOptHiveTable table = (RelOptHiveTable) scan.getTable();
  4.  List<String> bucketCols = table.getHiveTableMD().getBucketCols();
  5.  if (bucketCols != null && !bucketCols.isEmpty()) {
  6.   splitCount = table.getHiveTableMD().getNumBuckets();
  7.  } else {
  8.   splitCount = splitCountRepartition(scan, mq);
  9.   if (splitCount == null) {
  10.    throw new RuntimeException("Could not get split count for table: "
  11.      + scan.getTable().getQualifiedName());
  12.   }
  13.  }
  14.  return splitCount;
  15. }

代码示例来源：origin: apache/drill

  1. public boolean checkBucketedTable(Table tbl, ParseContext pGraphContext,
  2.   PrunedPartitionList prunedParts) throws SemanticException {
  3.  final int numBuckets = tbl.getNumBuckets();
  4.  if (numBuckets <= 0) {
  5.   return false;

代码示例来源：origin: apache/hive

  1. public boolean checkBucketedTable(Table tbl, ParseContext pGraphContext,
  2.   PrunedPartitionList prunedParts) throws SemanticException {
  3.  final int numBuckets = tbl.getNumBuckets();
  4.  if (numBuckets <= 0) {
  5.   return false;

代码示例来源：origin: apache/drill

  1. public Integer splitCount(HiveTableScan scan, RelMetadataQuery mq) {
  2.  Integer splitCount;
  3.  RelOptHiveTable table = (RelOptHiveTable) scan.getTable();
  4.  List<String> bucketCols = table.getHiveTableMD().getBucketCols();
  5.  if (bucketCols != null && !bucketCols.isEmpty()) {
  6.   splitCount = table.getHiveTableMD().getNumBuckets();
  7.  } else {
  8.   splitCount = splitCountRepartition(scan, mq);
  9.   if (splitCount == null) {
  10.    throw new RuntimeException("Could not get split count for table: "
  11.      + scan.getTable().getQualifiedName());
  12.   }
  13.  }
  14.  return splitCount;
  15. }

代码示例来源：origin: apache/hive

  1. private boolean checkTable(Table table,
  2.   List<Integer> bucketPositionsDest,
  3.   List<Integer> sortPositionsDest,
  4.   List<Integer> sortOrderDest,
  5.   int numBucketsDest) {
  6.  // The bucketing and sorting positions should exactly match
  7.  int numBuckets = table.getNumBuckets();
  8.  if (numBucketsDest != numBuckets) {
  9.   return false;
  10.  }
  11.  List<Integer> tableBucketPositions =
  12.    getBucketPositions(table.getBucketCols(), table.getCols());
  13.  List<Integer> sortPositions =
  14.    getSortPositions(table.getSortCols(), table.getCols());
  15.  List<Integer> sortOrder =
  16.    getSortOrder(table.getSortCols(), table.getCols());
  17.  return bucketPositionsDest.equals(tableBucketPositions) &&
  18.    sortPositionsDest.equals(sortPositions) &&
  19.    sortOrderDest.equals(sortOrder);
  20. }

代码示例来源：origin: apache/drill

  1. private boolean checkTable(Table table,
  2.   List<Integer> bucketPositionsDest,
  3.   List<Integer> sortPositionsDest,
  4.   List<Integer> sortOrderDest,
  5.   int numBucketsDest) {
  6.  // The bucketing and sorting positions should exactly match
  7.  int numBuckets = table.getNumBuckets();
  8.  if (numBucketsDest != numBuckets) {
  9.   return false;
  10.  }
  11.  List<Integer> tableBucketPositions =
  12.    getBucketPositions(table.getBucketCols(), table.getCols());
  13.  List<Integer> sortPositions =
  14.    getSortPositions(table.getSortCols(), table.getCols());
  15.  List<Integer> sortOrder =
  16.    getSortOrder(table.getSortCols(), table.getCols());
  17.  return bucketPositionsDest.equals(tableBucketPositions) &&
  18.    sortPositionsDest.equals(sortPositions) &&
  19.    sortOrderDest.equals(sortOrder);
  20. }

代码示例来源：origin: apache/hive

  1. if (isBucketed) {
  2.  bucketColsList.add(table.getBucketCols());
  3.  numBuckets = table.getNumBuckets();
  4.  List<String> sortCols = new ArrayList<String>();
  5.  for (Order colSortOrder : table.getSortCols()) {

代码示例来源：origin: apache/hive

  1. private void genPartnCols(String dest, Operator input, QB qb,
  2.   TableDesc table_desc, Table dest_tab, SortBucketRSCtx ctx) throws SemanticException {
  3.  boolean enforceBucketing = false;
  4.  ArrayList<ExprNodeDesc> partnColsNoConvert = new ArrayList<ExprNodeDesc>();
  5.  if ((dest_tab.getNumBuckets() > 0)) {
  6.   enforceBucketing = true;
  7.   if (updating(dest) || deleting(dest)) {
  8.    partnColsNoConvert = getPartitionColsFromBucketColsForUpdateDelete(input, false);
  9.   } else {
  10.    partnColsNoConvert = getPartitionColsFromBucketCols(dest, qb, dest_tab, table_desc, input,
  11.      false);
  12.   }
  13.  }
  14.  if ((dest_tab.getSortCols() != null) &&
  15.    (dest_tab.getSortCols().size() > 0)) {
  16.   if (!enforceBucketing) {
  17.    throw new SemanticException(ErrorMsg.TBL_SORTED_NOT_BUCKETED.getErrorCodedMsg(dest_tab.getCompleteName()));
  18.   }
  19.   else {
  20.    if(!enforceBucketing) {
  21.     partnColsNoConvert = getSortCols(dest, qb, dest_tab, table_desc, input, false);
  22.    }
  23.   }
  24.   enforceBucketing = true;
  25.  }
  26.  if (enforceBucketing) {
  27.   ctx.setPartnCols(partnColsNoConvert);
  28.  }
  29. }

代码示例来源：origin: apache/hive

  1. private DynamicPartitionCtx checkDynPart(QB qb, QBMetaData qbm, Table dest_tab,
  2.                      Map<String, String> partSpec, String dest) throws SemanticException {
  3.  List<FieldSchema> parts = dest_tab.getPartitionKeys();
  4.  if (parts == null || parts.isEmpty()) {
  5.   return null; // table is not partitioned
  6.  }
  7.  if (partSpec == null || partSpec.size() == 0) { // user did NOT specify partition
  8.   throw new SemanticException(generateErrorMessage(qb.getParseInfo().getDestForClause(dest),
  9.     ErrorMsg.NEED_PARTITION_ERROR.getMsg()));
  10.  }
  11.  DynamicPartitionCtx dpCtx = qbm.getDPCtx(dest);
  12.  if (dpCtx == null) {
  13.   dest_tab.validatePartColumnNames(partSpec, false);
  14.   dpCtx = new DynamicPartitionCtx(partSpec,
  15.     conf.getVar(HiveConf.ConfVars.DEFAULTPARTITIONNAME),
  16.     conf.getIntVar(HiveConf.ConfVars.DYNAMICPARTITIONMAXPARTSPERNODE));
  17.   qbm.setDPCtx(dest, dpCtx);
  18.  }
  19.  if (!HiveConf.getBoolVar(conf, HiveConf.ConfVars.DYNAMICPARTITIONING)) { // allow DP
  20.   throw new SemanticException(generateErrorMessage(qb.getParseInfo().getDestForClause(dest),
  21.     ErrorMsg.DYNAMIC_PARTITION_DISABLED.getMsg()));
  22.  }
  23.  if ((dest_tab.getNumBuckets() > 0)) {
  24.   dpCtx.setNumBuckets(dest_tab.getNumBuckets());
  25.  }
  26.  return dpCtx;
  27. }

代码示例来源：origin: apache/drill

  1. private void checkAcidConstraints(QB qb, TableDesc tableDesc,
  2.                  Table table) throws SemanticException {
  3.  String tableName = tableDesc.getTableName();
  4.  if (!qb.getParseInfo().isInsertIntoTable(tableName)) {
  5.   LOG.debug("Couldn't find table " + tableName + " in insertIntoTable");
  6.   throw new SemanticException(ErrorMsg.NO_INSERT_OVERWRITE_WITH_ACID.getMsg());
  7.  }
  8.  /*
  9.  LOG.info("Modifying config values for ACID write");
  10.  conf.setBoolVar(ConfVars.HIVEOPTREDUCEDEDUPLICATION, true);
  11.  conf.setIntVar(ConfVars.HIVEOPTREDUCEDEDUPLICATIONMINREDUCER, 1);
  12.  These props are now enabled elsewhere (see commit diffs).  It would be better instead to throw
  13.  if they are not set.  For exmaple, if user has set hive.optimize.reducededuplication=false for
  14.  some reason, we'll run a query contrary to what they wanted...  But throwing now would be
  15.  backwards incompatible.
  16.  */
  17.  conf.set(AcidUtils.CONF_ACID_KEY, "true");
  18.  if (table.getNumBuckets() < 1) {
  19.   throw new SemanticException(ErrorMsg.ACID_OP_ON_NONACID_TABLE, table.getTableName());
  20.  }
  21.  if (table.getSortCols() != null && table.getSortCols().size() > 0) {
  22.   throw new SemanticException(ErrorMsg.ACID_NO_SORTED_BUCKETS, table.getTableName());
  23.  }
  24. }

代码示例来源：origin: apache/drill

  1. private void genPartnCols(String dest, Operator input, QB qb,
  2.   TableDesc table_desc, Table dest_tab, SortBucketRSCtx ctx) throws SemanticException {
  3.  boolean enforceBucketing = false;
  4.  ArrayList<ExprNodeDesc> partnColsNoConvert = new ArrayList<ExprNodeDesc>();
  5.  if ((dest_tab.getNumBuckets() > 0)) {
  6.   enforceBucketing = true;
  7.   if (updating(dest) || deleting(dest)) {
  8.    partnColsNoConvert = getPartitionColsFromBucketColsForUpdateDelete(input, false);
  9.   } else {
  10.    partnColsNoConvert = getPartitionColsFromBucketCols(dest, qb, dest_tab, table_desc, input,
  11.      false);
  12.   }
  13.  }
  14.  if ((dest_tab.getSortCols() != null) &&
  15.    (dest_tab.getSortCols().size() > 0)) {
  16.   if (!enforceBucketing && !dest_tab.isIndexTable()) {
  17.    throw new SemanticException(ErrorMsg.TBL_SORTED_NOT_BUCKETED.getErrorCodedMsg(dest_tab.getCompleteName()));
  18.   }
  19.   else {
  20.    if(!enforceBucketing) {
  21.     partnColsNoConvert = getSortCols(dest, qb, dest_tab, table_desc, input, false);
  22.    }
  23.   }
  24.   enforceBucketing = true;
  25.  }
  26.  if (enforceBucketing) {
  27.   ctx.setPartnCols(partnColsNoConvert);
  28.  }
  29. }

代码示例来源：origin: apache/hive

  1. lbLevels = lbCtx == null ? 0 : lbCtx.calculateListBucketingLevel();
  2. int numBuckets = (conf.getTable() != null) ? conf.getTable().getNumBuckets()
  3.   : (dpCtx != null ? dpCtx.getNumBuckets() : 0);
  4. MissingBucketsContext mbc = new MissingBucketsContext(

代码示例来源：origin: apache/hive

  1. private void alterPartitionSpecInMemory(Table tbl,
  2.   Map<String, String> partSpec,
  3.   org.apache.hadoop.hive.metastore.api.Partition tpart,
  4.   boolean inheritTableSpecs,
  5.   String partPath) throws HiveException, InvalidOperationException {
  6.  LOG.debug("altering partition for table " + tbl.getTableName() + " with partition spec : "
  7.    + partSpec);
  8.  if (inheritTableSpecs) {
  9.   tpart.getSd().setOutputFormat(tbl.getTTable().getSd().getOutputFormat());
  10.   tpart.getSd().setInputFormat(tbl.getTTable().getSd().getInputFormat());
  11.   tpart.getSd().getSerdeInfo().setSerializationLib(tbl.getSerializationLib());
  12.   tpart.getSd().getSerdeInfo().setParameters(
  13.     tbl.getTTable().getSd().getSerdeInfo().getParameters());
  14.   tpart.getSd().setBucketCols(tbl.getBucketCols());
  15.   tpart.getSd().setNumBuckets(tbl.getNumBuckets());
  16.   tpart.getSd().setSortCols(tbl.getSortCols());
  17.  }
  18.  if (partPath == null || partPath.trim().equals("")) {
  19.   throw new HiveException("new partition path should not be null or empty.");
  20.  }
  21.  tpart.getSd().setLocation(partPath);
  22. }

代码示例来源：origin: apache/drill

  1. private void alterPartitionSpecInMemory(Table tbl,
  2.   Map<String, String> partSpec,
  3.   org.apache.hadoop.hive.metastore.api.Partition tpart,
  4.   boolean inheritTableSpecs,
  5.   String partPath) throws HiveException, InvalidOperationException {
  6.  LOG.debug("altering partition for table " + tbl.getTableName() + " with partition spec : "
  7.    + partSpec);
  8.  if (inheritTableSpecs) {
  9.   tpart.getSd().setOutputFormat(tbl.getTTable().getSd().getOutputFormat());
  10.   tpart.getSd().setInputFormat(tbl.getTTable().getSd().getInputFormat());
  11.   tpart.getSd().getSerdeInfo().setSerializationLib(tbl.getSerializationLib());
  12.   tpart.getSd().getSerdeInfo().setParameters(
  13.     tbl.getTTable().getSd().getSerdeInfo().getParameters());
  14.   tpart.getSd().setBucketCols(tbl.getBucketCols());
  15.   tpart.getSd().setNumBuckets(tbl.getNumBuckets());
  16.   tpart.getSd().setSortCols(tbl.getSortCols());
  17.  }
  18.  if (partPath == null || partPath.trim().equals("")) {
  19.   throw new HiveException("new partition path should not be null or empty.");
  20.  }
  21.  tpart.getSd().setLocation(partPath);
  22. }

内容来源于网络，如有侵权，请联系作者删除！

Table

相关文章

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多

    Table类方法

    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com