kafka部署卡在vagrant集群上

zaqlnxep  于 2021-06-08  发布在  Kafka
关注(0)|答案(2)|浏览(410)

我正在使用最新的dcos发行版（0.8.0），并使用4个私有代理节点、一个公共代理节点、一个主节点和一个引导机构建了一个本地vagrant集群

vagrant up m1 a1 a2 a3 a4 p1 boot

然后，我使用以下命令以默认配置安装了kafka：

dcos package install kafka

在检查了马拉松日志之后，mesos似乎拒绝了Kafka的资源报价，因此无法部署所有三个代理。我怀疑问题出在提供资源的角色和期望的Kafka角色之间不匹配。
聚合群集统计信息：cpu:18共享内存：7gib
以下是马拉松日志的快照：

s.offer.OfferEvaluator: Satisfying resource requirement: name: "cpus" type: SCALAR scalar { value: 1.0 } role: "kafka-role" reservation { principal: "kafka-principal" labels { labels { key: "resource_id" value: "" } } }
with resource: name: "cpus" type: SCALAR scalar { value: 1.0 } role: "*"
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Reserves Resource
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Fulfilled resource: name: "cpus" type: SCALAR scalar { value: 1.0 } role: "kafka-role" reservation { principal: "kafka-principal" labels { labels { key: "resource_id" value: "f8468a5a-8166-4d5d-a594-f197a8b356cd" } } }
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.MesosResourcePool: Retrieving resource for reservation
WARN  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Failed to satisfy resource requirement: name: "mem" type: SCALAR scalar { value: 2304.0 } role: "kafka-role" reservation { principal: "kafka-principal" labels { labels { key: "resource_id" value: "" } } }
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.MesosResourcePool: Retrieving resource for reservation
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Satisfying resource requirement: name: "cpus" type: SCALAR scalar { value: 0.5 } role: "kafka-role" reservation { principal: "kafka-principal" labels { labels { key: "resource_id" value: "" } } }
with resource: name: "cpus" type: SCALAR scalar { value: 0.5 } role: "*"
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Reserves Resource
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Fulfilled resource: name: "cpus" type: SCALAR scalar { value: 0.5 } role: "kafka-role" reservation { principal: "kafka-principal" labels { labels { key: "resource_id" value: "37ba47a3-7c6e-47a2-972c-d35d130ae8df" } } }
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.MesosResourcePool: Retrieving resource for reservation
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Satisfying resource requirement: name: "mem" type: SCALAR scalar { value: 256.0 } role: "kafka-role" reservation { principal: "kafka-principal" labels { labels { key: "resource_id" value: "" } } }
with resource: name: "mem" type: SCALAR scalar { value: 256.0 } role: "*"
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Reserves Resource
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Fulfilled resource: name: "mem" type: SCALAR scalar { value: 256.0 } role: "kafka-role" reservation { principal: "kafka-principal" labels { labels { key: "resource_id" value: "8f937e43-c80a-4434-a858-08e6bdb02cc2" } } }
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.MesosResourcePool: Retrieving resource for reservation
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Satisfying resource requirement: name: "cpus" type: SCALAR scalar { value: 1.0 } role: "kafka-role" reservation { principal: "kafka-principal" labels { labels { key: "resource_id" value: "" } } }
with resource: name: "cpus" type: SCALAR scalar { value: 1.0 } role: "*"
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Reserves Resource
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Fulfilled resource: name: "cpus" type: SCALAR scalar { value: 1.0 } role: "kafka-role" reservation { principal: "kafka-principal" labels { labels { key: "resource_id" value: "b568cfaa-c216-4479-9853-5bc4e6ee2d9e" } } }
INFO  [2016-08-02 16:28:08,919] org.apache.mesos.offer.MesosResourcePool: Retrieving resource for reservation
WARN  [2016-08-02 16:28:08,919] org.apache.mesos.offer.OfferEvaluator: Failed to satisfy resource requirement: name: "mem" type: SCALAR scalar { value: 2304.0 } role: "kafka-role" reservation { principal: "kafka-principal" labels { labels { key: "resource_id" value: "" } } }
WARN  [2016-08-02 16:28:08,919] org.apache.mesos.scheduler.plan.DefaultStageScheduler: Unable to find any offers which fulfill requirement provided by block broker-0: org.apache.mesos.offer.OfferRequirement@2a28d9e[avoidAgents=[],colocateAgents=[],taskRequirements=[org.apache.mesos.offer.TaskRequirement@5c8e5b0f],executorRequirement=org.apache.mesos.offer.ExecutorRequirement@6179902f[executorInfo=executor_id {
  value: ""
}
resources {
  name: "cpus"
  type: SCALAR
  scalar {
    value: 0.5
  }
  role: "kafka-role"
  reservation {
    principal: "kafka-principal"
    labels {
      labels {
        key: "resource_id"
        value: ""
      }
    }
  }
}
resources {
  name: "mem"
  type: SCALAR
  scalar {
    value: 256.0
  }
  role: "kafka-role"
  reservation {
    principal: "kafka-principal"
    labels {
      labels {
        key: "resource_id"
        value: ""
      }
    }
  }
}
command {
  uris {
    value: "https://downloads.mesosphere.com/kafka/assets/jre-8u91-linux-x64.tar.gz"
  }
  uris {
    value: "https://downloads.mesosphere.com/kafka/assets/kafka_2.11-0.10.0.0.tgz"
  }
  uris {
    value: "https://downloads.mesosphere.com/kafka/assets/1.1.9-0.10.0.0/overrider.zip"
  }
  uris {
    value: "https://downloads.mesosphere.com/kafka/assets/1.1.9-0.10.0.0/executor.zip"
  }
  environment {
    variables {
      name: "JAVA_HOME"
      value: "jre1.8.0_91"
    }
  }
  value: "./executor/bin/kafka-executor -Dlogback.configurationFile=executor/conf/logback.xml"
}
framework_id {
  value: "2601d122-2b59-4823-83f8-84351376d53c-0001"
}
name: "broker-0"
,resourceRequirements=[org.apache.mesos.offer.ResourceRequirement@39ba5849, org.apache.mesos.offer.ResourceRequirement@2f62ec9d]]]
INFO  [2016-08-02 16:28:08,919] com.mesosphere.dcos.kafka.plan.KafkaUpdateBlock: broker-0: changed status from: Pending to: Pending
INFO  [2016-08-02 16:28:08,920] com.mesosphere.dcos.kafka.scheduler.KafkaRepairScheduler: Missing brokerIds: []
INFO  [2016-08-02 16:28:08,921] org.apache.mesos.offer.ResourceCleaner: 4 offers with 16 resources => 0 destroy and 0 unreserve operations
INFO  [2016-08-02 16:28:08,921] com.mesosphere.dcos.kafka.scheduler.KafkaScheduler: Accepted 0 of 4 offers: []
INFO  [2016-08-02 16:28:08,921] com.mesosphere.dcos.kafka.scheduler.KafkaScheduler: Declining offer: 2601d122-2b59-4823-83f8-84351376d53c-O1343
INFO  [2016-08-02 16:28:08,921] com.mesosphere.dcos.kafka.scheduler.KafkaScheduler: Declining offer: 2601d122-2b59-4823-83f8-84351376d53c-O1344
INFO  [2016-08-02 16:28:08,921] com.mesosphere.dcos.kafka.scheduler.KafkaScheduler: Declining offer: 2601d122-2b59-4823-83f8-84351376d53c-O1345
INFO  [2016-08-02 16:28:08,921] com.mesosphere.dcos.kafka.scheduler.KafkaScheduler: Declining offer: 2601d122-2b59-4823-83f8-84351376d53c-O1346
INFO  [2016-08-02 16:28:14,928]
apache-kafkamesosdcosvagrant

2条答案

按热度按时间
b1payxdu

b1payxdu1#

问题确实如日志所述：没有足够的内存来使用默认设置安排Kafka。我可以使用默认的vagrant配置在dcos vagrant上部署kafka，并使用以下kafka配置在三个代理节点上部署：

{
    "brokers": {
        "count": 1,
        "mem": 512,
        "disk": 1000
    }
}
赞(0）分享  回复(0）举报  2021-06-09
jqjz2hbq

jqjz2hbq2#

dcos1.8中的kafka框架是一种有状态服务。它专门设计用于为每个代理分配存储，并维护代理示例与该存储之间的链接（因此，不会将代理0重新部署到没有最初为代理0分配存储的主机上）。我的经验是，当不满足存储限制时，围绕资源提供的消息不是很清楚。默认情况下，包将为每个代理请求5gb的专用存储空间。
我对vagrant没有太多的经验，但可能是vagrant没有为dc/os提供必要的“存储可用性”来满足代理资源请求。

赞(0）分享  回复(0）举报  2021-06-09
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com