org.apache.hadoop.yarn.util.resource.Resources.min()方法的使用及代码示例

x33g5p2x  于2022-01-29 转载在 其他  
字(17.4k)|赞(0)|评价(0)|浏览(142)

本文整理了Java中org.apache.hadoop.yarn.util.resource.Resources.min方法的一些代码示例，展示了Resources.min的具体用法。这些代码示例主要来源于Github/Stackoverflow/Maven等平台，是从一些精选项目中提取出来的代码，具有较强的参考意义，能在一定程度帮忙到你。Resources.min方法的具体详情如下：
包路径：org.apache.hadoop.yarn.util.resource.Resources
类名称：Resources
方法名：min

Resources.min介绍

暂无

代码示例

代码示例来源：origin: com.github.jiayuhan-it/hadoop-yarn-server-resourcemanager

private Resource getCurrentLimitResource(String nodeLabel,
  Resource clusterResource, ResourceLimits currentResourceLimits) {
 /*
  * Current limit resource: For labeled resource: limit = queue-max-resource
  * (TODO, this part need update when we support labeled-limit) For
  * non-labeled resource: limit = min(queue-max-resource,
  * limit-set-by-parent)
  */
 Resource queueMaxResource =
   Resources.multiplyAndNormalizeDown(resourceCalculator,
     labelManager.getResourceByLabel(nodeLabel, clusterResource),
     queueCapacities.getAbsoluteMaximumCapacity(nodeLabel), minimumAllocation);
 if (nodeLabel.equals(RMNodeLabelsManager.NO_LABEL)) {
  return Resources.min(resourceCalculator, clusterResource,
    queueMaxResource, currentResourceLimits.getLimit());
 }
 return queueMaxResource;
}

代码示例来源：origin: ch.cern.hadoop/hadoop-yarn-server-resourcemanager

private Resource getCurrentLimitResource(String nodeLabel,
  Resource clusterResource, ResourceLimits currentResourceLimits) {
 /*
  * Current limit resource: For labeled resource: limit = queue-max-resource
  * (TODO, this part need update when we support labeled-limit) For
  * non-labeled resource: limit = min(queue-max-resource,
  * limit-set-by-parent)
  */
 Resource queueMaxResource =
   Resources.multiplyAndNormalizeDown(resourceCalculator,
     labelManager.getResourceByLabel(nodeLabel, clusterResource),
     queueCapacities.getAbsoluteMaximumCapacity(nodeLabel), minimumAllocation);
 if (nodeLabel.equals(RMNodeLabelsManager.NO_LABEL)) {
  return Resources.min(resourceCalculator, clusterResource,
    queueMaxResource, currentResourceLimits.getLimit());
 }
 return queueMaxResource;
}

代码示例来源：origin: com.github.jiayuhan-it/hadoop-yarn-server-resourcemanager

Resource offer(Resource avail, ResourceCalculator rc,
  Resource clusterResource) {
 Resource absMaxCapIdealAssignedDelta = Resources.componentwiseMax(
         Resources.subtract(maxCapacity, idealAssigned),
         Resource.newInstance(0, 0));
 // remain = avail - min(avail, (max - assigned), (current + pending - assigned))
 Resource accepted = 
   Resources.min(rc, clusterResource, 
     absMaxCapIdealAssignedDelta,
   Resources.min(rc, clusterResource, avail, Resources.subtract(
     Resources.add(current, pending), idealAssigned)));
 Resource remain = Resources.subtract(avail, accepted);
 Resources.addTo(idealAssigned, accepted);
 return remain;
}

代码示例来源：origin: ch.cern.hadoop/hadoop-yarn-server-resourcemanager

Resource offer(Resource avail, ResourceCalculator rc,
  Resource clusterResource) {
 Resource absMaxCapIdealAssignedDelta = Resources.componentwiseMax(
         Resources.subtract(maxCapacity, idealAssigned),
         Resource.newInstance(0, 0));
 // remain = avail - min(avail, (max - assigned), (current + pending - assigned))
 Resource accepted = 
   Resources.min(rc, clusterResource, 
     absMaxCapIdealAssignedDelta,
   Resources.min(rc, clusterResource, avail, Resources.subtract(
     Resources.add(current, pending), idealAssigned)));
 Resource remain = Resources.subtract(avail, accepted);
 Resources.addTo(idealAssigned, accepted);
 return remain;
}

代码示例来源：origin: org.apache.hadoop/hadoop-yarn-server-resourcemanager

private Resource getCurrentLimitResource(String nodePartition,
  Resource clusterResource, ResourceLimits currentResourceLimits,
  SchedulingMode schedulingMode) {
 if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) {
  /*
   * Current limit resource: For labeled resource: limit = queue-max-resource
   * (TODO, this part need update when we support labeled-limit) For
   * non-labeled resource: limit = min(queue-max-resource,
   * limit-set-by-parent)
   */
  Resource queueMaxResource =
    getQueueMaxResource(nodePartition);
  return Resources.min(resourceCalculator, clusterResource,
    queueMaxResource, currentResourceLimits.getLimit());
 } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY) {
  // When we doing non-exclusive resource allocation, maximum capacity of
  // all queues on this label equals to total resource with the label.
  return labelManager.getResourceByLabel(nodePartition, clusterResource);
 }
 
 return Resources.none();
}

代码示例来源：origin: org.apache.hadoop/hadoop-yarn-server-resourcemanager

private void updateCurrentResourceLimits(
  ResourceLimits currentResourceLimits, Resource clusterResource) {
 // TODO: need consider non-empty node labels when resource limits supports
 // node labels
 // Even if ParentQueue will set limits respect child's max queue capacity,
 // but when allocating reserved container, CapacityScheduler doesn't do
 // this. So need cap limits by queue's max capacity here.
 this.cachedResourceLimitsForHeadroom =
   new ResourceLimits(currentResourceLimits.getLimit());
 Resource queueMaxResource = getEffectiveMaxCapacityDown(
   RMNodeLabelsManager.NO_LABEL, minimumAllocation);
 this.cachedResourceLimitsForHeadroom.setLimit(Resources.min(
   resourceCalculator, clusterResource, queueMaxResource,
   currentResourceLimits.getLimit()));
}

代码示例来源：origin: org.apache.hadoop/hadoop-yarn-server-resourcemanager

/**
 * This method is visible to allow sub-classes to override the behavior,
 * specifically for federation purposes we do not want to cap resources as it
 * is done here.
 *
 * @param rc the {@code ResourceCalculator} to be used
 * @param clusterResource the total cluster resources
 * @param offered the resources offered to this queue
 * @return the amount of resources accepted after considering max and
 *         deducting assigned.
 */
protected Resource filterByMaxDeductAssigned(ResourceCalculator rc,
  Resource clusterResource, Resource offered) {
 if (null == children || children.isEmpty()) {
  Resource maxOfGuranteedAndUsedDeductAssigned = Resources.subtract(
    Resources.max(rc, clusterResource, getUsed(), getGuaranteed()),
    idealAssigned);
  maxOfGuranteedAndUsedDeductAssigned = Resources.max(rc, clusterResource,
    maxOfGuranteedAndUsedDeductAssigned, Resources.none());
  offered = Resources.min(rc, clusterResource, offered,
    maxOfGuranteedAndUsedDeductAssigned);
 }
 return offered;
}

代码示例来源：origin: ch.cern.hadoop/hadoop-yarn-server-resourcemanager

private boolean isStarved(Resource share) {
  Resource desiredShare = Resources.min(scheduler.getResourceCalculator(),
    scheduler.getClusterResource(), share, getDemand());
  return Resources.lessThan(scheduler.getResourceCalculator(),
    scheduler.getClusterResource(), getResourceUsage(), desiredShare);
 }
}

代码示例来源：origin: com.github.jiayuhan-it/hadoop-yarn-server-resourcemanager

private boolean isStarved(Resource share) {
  Resource desiredShare = Resources.min(scheduler.getResourceCalculator(),
    scheduler.getClusterResource(), share, getDemand());
  return Resources.lessThan(scheduler.getResourceCalculator(),
    scheduler.getClusterResource(), getResourceUsage(), desiredShare);
 }
}

代码示例来源：origin: com.github.jiayuhan-it/hadoop-yarn-server-resourcemanager

private void updateCurrentResourceLimits(
  ResourceLimits currentResourceLimits, Resource clusterResource) {
 // TODO: need consider non-empty node labels when resource limits supports
 // node labels
 // Even if ParentQueue will set limits respect child's max queue capacity,
 // but when allocating reserved container, CapacityScheduler doesn't do
 // this. So need cap limits by queue's max capacity here.
 this.cachedResourceLimitsForHeadroom = new ResourceLimits(currentResourceLimits.getLimit());
 Resource queueMaxResource =
   Resources.multiplyAndNormalizeDown(resourceCalculator, labelManager
     .getResourceByLabel(RMNodeLabelsManager.NO_LABEL, clusterResource),
     queueCapacities
       .getAbsoluteMaximumCapacity(RMNodeLabelsManager.NO_LABEL),
     minimumAllocation);
 this.cachedResourceLimitsForHeadroom.setLimit(Resources.min(resourceCalculator,
   clusterResource, queueMaxResource, currentResourceLimits.getLimit()));
}

代码示例来源：origin: ch.cern.hadoop/hadoop-yarn-server-resourcemanager

private void updateCurrentResourceLimits(
  ResourceLimits currentResourceLimits, Resource clusterResource) {
 // TODO: need consider non-empty node labels when resource limits supports
 // node labels
 // Even if ParentQueue will set limits respect child's max queue capacity,
 // but when allocating reserved container, CapacityScheduler doesn't do
 // this. So need cap limits by queue's max capacity here.
 this.cachedResourceLimitsForHeadroom = new ResourceLimits(currentResourceLimits.getLimit());
 Resource queueMaxResource =
   Resources.multiplyAndNormalizeDown(resourceCalculator, labelManager
     .getResourceByLabel(RMNodeLabelsManager.NO_LABEL, clusterResource),
     queueCapacities
       .getAbsoluteMaximumCapacity(RMNodeLabelsManager.NO_LABEL),
     minimumAllocation);
 this.cachedResourceLimitsForHeadroom.setLimit(Resources.min(resourceCalculator,
   clusterResource, queueMaxResource, currentResourceLimits.getLimit()));
}

代码示例来源：origin: com.github.jiayuhan-it/hadoop-yarn-server-resourcemanager

private ResourceLimits getResourceLimitsOfChild(CSQueue child,
  Resource clusterResource, ResourceLimits parentLimits) {
 // Set resource-limit of a given child, child.limit =
 // min(my.limit - my.used + child.used, child.max)
 // Parent available resource = parent-limit - parent-used-resource
 Resource parentMaxAvailableResource =
   Resources.subtract(parentLimits.getLimit(), getUsedResources());
 // Child's limit = parent-available-resource + child-used
 Resource childLimit =
   Resources.add(parentMaxAvailableResource, child.getUsedResources());
 // Get child's max resource
 Resource childConfiguredMaxResource =
   Resources.multiplyAndNormalizeDown(resourceCalculator, labelManager
     .getResourceByLabel(RMNodeLabelsManager.NO_LABEL, clusterResource),
     child.getAbsoluteMaximumCapacity(), minimumAllocation);
 // Child's limit should be capped by child configured max resource
 childLimit =
   Resources.min(resourceCalculator, clusterResource, childLimit,
     childConfiguredMaxResource);
 // Normalize before return
 childLimit =
   Resources.roundDown(resourceCalculator, childLimit, minimumAllocation);
 return new ResourceLimits(childLimit);
}

代码示例来源：origin: org.apache.hadoop/hadoop-yarn-server-resourcemanager

public void assignPreemption(float scalingFactor, ResourceCalculator rc,
  Resource clusterResource) {
 Resource usedDeductKillable = Resources.subtract(getUsed(), killable);
 Resource totalResource = Resources.add(getUsed(), pending);
 // The minimum resource that we need to keep for a queue is:
 // max(idealAssigned, min(used + pending, guaranteed)).
 //
 // Doing this because when we calculate ideal allocation doesn't consider
 // reserved resource, ideal-allocation calculated could be less than
 // guaranteed and total. We should avoid preempt from a queue if it is
 // already
 // <= its guaranteed resource.
 Resource minimumQueueResource = Resources.max(rc, clusterResource,
   Resources.min(rc, clusterResource, totalResource, getGuaranteed()),
   idealAssigned);
 if (Resources.greaterThan(rc, clusterResource, usedDeductKillable,
   minimumQueueResource)) {
  toBePreempted = Resources.multiply(
    Resources.subtract(usedDeductKillable, minimumQueueResource),
    scalingFactor);
 } else {
  toBePreempted = Resources.none();
 }
}

代码示例来源：origin: ch.cern.hadoop/hadoop-yarn-server-resourcemanager

private ResourceLimits getResourceLimitsOfChild(CSQueue child,
  Resource clusterResource, ResourceLimits parentLimits) {
 // Set resource-limit of a given child, child.limit =
 // min(my.limit - my.used + child.used, child.max)
 // Parent available resource = parent-limit - parent-used-resource
 Resource parentMaxAvailableResource =
   Resources.subtract(parentLimits.getLimit(), getUsedResources());
 // Child's limit = parent-available-resource + child-used
 Resource childLimit =
   Resources.add(parentMaxAvailableResource, child.getUsedResources());
 // Get child's max resource
 Resource childConfiguredMaxResource =
   Resources.multiplyAndNormalizeDown(resourceCalculator, labelManager
     .getResourceByLabel(RMNodeLabelsManager.NO_LABEL, clusterResource),
     child.getAbsoluteMaximumCapacity(), minimumAllocation);
 // Child's limit should be capped by child configured max resource
 childLimit =
   Resources.min(resourceCalculator, clusterResource, childLimit,
     childConfiguredMaxResource);
 // Normalize before return
 childLimit =
   Resources.roundDown(resourceCalculator, childLimit, minimumAllocation);
 return new ResourceLimits(childLimit);
}

代码示例来源：origin: org.apache.hadoop/hadoop-yarn-server-resourcemanager

private ResourceLimits getResourceLimitsOfChild(CSQueue child,
  Resource clusterResource, Resource parentLimits,
  String nodePartition) {
 // Set resource-limit of a given child, child.limit =
 // min(my.limit - my.used + child.used, child.max)
 // Parent available resource = parent-limit - parent-used-resource
 Resource parentMaxAvailableResource = Resources.subtract(
   parentLimits, queueUsage.getUsed(nodePartition));
 // Deduct killable from used
 Resources.addTo(parentMaxAvailableResource,
   getTotalKillableResource(nodePartition));
 // Child's limit = parent-available-resource + child-used
 Resource childLimit = Resources.add(parentMaxAvailableResource,
   child.getQueueResourceUsage().getUsed(nodePartition));
 // Get child's max resource
 Resource childConfiguredMaxResource = child
   .getEffectiveMaxCapacityDown(nodePartition, minimumAllocation);
 // Child's limit should be capped by child configured max resource
 childLimit =
   Resources.min(resourceCalculator, clusterResource, childLimit,
     childConfiguredMaxResource);
 // Normalize before return
 childLimit =
   Resources.roundDown(resourceCalculator, childLimit, minimumAllocation);
 return new ResourceLimits(childLimit);
}

代码示例来源：origin: org.apache.hadoop/hadoop-yarn-server-resourcemanager

untouchableExtra = Resources.subtract(extra, childrensPreemptable);
preemptableExtra = Resources.min(rc, totalPartitionResource,
  childrensPreemptable, extra);

代码示例来源：origin: org.apache.hadoop/hadoop-yarn-server-resourcemanager

private void calculateToBePreemptedResourcePerApp(Resource clusterResource,
  TreeSet<TempAppPerPartition> orderedApps, Resource preemptionLimit) {
 for (TempAppPerPartition tmpApp : orderedApps) {
  if (Resources.lessThanOrEqual(rc, clusterResource, preemptionLimit,
    Resources.none())
    || Resources.lessThanOrEqual(rc, clusterResource, tmpApp.getUsed(),
      Resources.none())) {
   continue;
  }
  Resource preemtableFromApp = Resources.subtract(tmpApp.getUsed(),
    tmpApp.idealAssigned);
  Resources.subtractFromNonNegative(preemtableFromApp, tmpApp.selected);
  Resources.subtractFromNonNegative(preemtableFromApp, tmpApp.getAMUsed());
  if (context.getIntraQueuePreemptionOrderPolicy()
     .equals(IntraQueuePreemptionOrderPolicy.USERLIMIT_FIRST)) {
   Resources.subtractFromNonNegative(preemtableFromApp,
    tmpApp.getFiCaSchedulerApp().getCSLeafQueue().getMinimumAllocation());
  }
  // Calculate toBePreempted from apps as follows:
  // app.preemptable = min(max(app.used - app.selected - app.ideal, 0),
  // intra_q_preemptable)
  tmpApp.toBePreempted = Resources.min(rc, clusterResource, Resources
    .max(rc, clusterResource, preemtableFromApp, Resources.none()),
    Resources.clone(preemptionLimit));
  preemptionLimit = Resources.subtractFromNonNegative(preemptionLimit,
    tmpApp.toBePreempted);
 }
}

代码示例来源：origin: org.apache.hadoop/hadoop-yarn-common

private void assertResourcesOperations(Resource clusterResource,
  Resource lhs, Resource rhs, boolean lessThan, boolean lessThanOrEqual,
  boolean greaterThan, boolean greaterThanOrEqual, Resource max,
  Resource min) {
 assertEquals("Less Than operation is wrongly calculated.", lessThan,
   Resources.lessThan(resourceCalculator, clusterResource, lhs, rhs));
 assertEquals(
   "Less Than Or Equal To operation is wrongly calculated.",
   lessThanOrEqual, Resources.lessThanOrEqual(resourceCalculator,
     clusterResource, lhs, rhs));
 assertEquals("Greater Than operation is wrongly calculated.",
   greaterThan,
   Resources.greaterThan(resourceCalculator, clusterResource, lhs, rhs));
 assertEquals(
   "Greater Than Or Equal To operation is wrongly calculated.",
   greaterThanOrEqual, Resources.greaterThanOrEqual(resourceCalculator,
     clusterResource, lhs, rhs));
 assertEquals("Max(value) Operation wrongly calculated.", max,
   Resources.max(resourceCalculator, clusterResource, lhs, rhs));
 assertEquals("Min(value) operation is wrongly calculated.", min,
   Resources.min(resourceCalculator, clusterResource, lhs, rhs));
}

代码示例来源：origin: ch.cern.hadoop/hadoop-yarn-common

private void assertResourcesOperations(Resource clusterResource,
  Resource lhs, Resource rhs, boolean lessThan, boolean lessThanOrEqual,
  boolean greaterThan, boolean greaterThanOrEqual, Resource max,
  Resource min) {
 Assert.assertEquals("Less Than operation is wrongly calculated.", lessThan,
   Resources.lessThan(resourceCalculator, clusterResource, lhs, rhs));
 Assert.assertEquals(
   "Less Than Or Equal To operation is wrongly calculated.",
   lessThanOrEqual, Resources.lessThanOrEqual(resourceCalculator,
     clusterResource, lhs, rhs));
 Assert.assertEquals("Greater Than operation is wrongly calculated.",
   greaterThan,
   Resources.greaterThan(resourceCalculator, clusterResource, lhs, rhs));
 Assert.assertEquals(
   "Greater Than Or Equal To operation is wrongly calculated.",
   greaterThanOrEqual, Resources.greaterThanOrEqual(resourceCalculator,
     clusterResource, lhs, rhs));
 Assert.assertEquals("Max(value) Operation wrongly calculated.", max,
   Resources.max(resourceCalculator, clusterResource, lhs, rhs));
 Assert.assertEquals("Min(value) operation is wrongly calculated.", min,
   Resources.min(resourceCalculator, clusterResource, lhs, rhs));
}

代码示例来源：origin: io.hops/hadoop-yarn-common

private void assertResourcesOperations(Resource clusterResource,
  Resource lhs, Resource rhs, boolean lessThan, boolean lessThanOrEqual,
  boolean greaterThan, boolean greaterThanOrEqual, Resource max,
  Resource min) {
 Assert.assertEquals("Less Than operation is wrongly calculated.", lessThan,
   Resources.lessThan(resourceCalculator, clusterResource, lhs, rhs));
 Assert.assertEquals(
   "Less Than Or Equal To operation is wrongly calculated.",
   lessThanOrEqual, Resources.lessThanOrEqual(resourceCalculator,
     clusterResource, lhs, rhs));
 Assert.assertEquals("Greater Than operation is wrongly calculated.",
   greaterThan,
   Resources.greaterThan(resourceCalculator, clusterResource, lhs, rhs));
 Assert.assertEquals(
   "Greater Than Or Equal To operation is wrongly calculated.",
   greaterThanOrEqual, Resources.greaterThanOrEqual(resourceCalculator,
     clusterResource, lhs, rhs));
 Assert.assertEquals("Max(value) Operation wrongly calculated.", max,
   Resources.max(resourceCalculator, clusterResource, lhs, rhs));
 Assert.assertEquals("Min(value) operation is wrongly calculated.", min,
   Resources.min(resourceCalculator, clusterResource, lhs, rhs));
}

内容来源于网络，如有侵权，请联系作者删除！

resources

相关文章

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新文章

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com