Docker堆栈中配置的HDFS容量错误

06odsfpq  于 2022-12-09  发布在  HDFS
关注(0)|答案(1)|浏览(166)

我使用的是Docker堆栈,它在同一台机器上实现了一个Hadoop Namenode、两个Datanodes、两个Node Manager、一个Resource Manager、一个History Server和其他技术。
我遇到了与HDFS UI中显示的HDFS配置容量相关的问题。
我使用的是一台容量为256GB的机器,并使用上面提到的两个数据节点实施。HDFS不是在两个节点之间分配总容量,而是通过为每个数据节点分配226.87GB来复制整个机器的容量。As you can see here
关于如何让HDFS显示正确的容量,您有什么想法吗?
下面是docker compose中实现上述hadoop技术的部分。

services:
  # Hadoop master
  namenode:
    image: bde2020/hadoop-namenode:2.0.0-hadoop3.2.1-java8
    container_name: namenode
    ports:
      - 9870:9870
      - 8020:8020
    volumes:
      - ./namenode/home/${ADMIN_NAME:?err}:/home/${ADMIN_NAME:?err}
      - ./namenode/hadoop-data:/hadoop-data
      - ./namenode/entrypoint.sh:/entrypoint.sh
      - hadoop-namenode:/hadoop/dfs/name
    env_file:
      - ./hadoop.env
      - .env
    networks:
      - hadoop

  resourcemanager:
    restart: always
    image: bde2020/hadoop-resourcemanager:2.0.0-hadoop3.2.1-java8
    container_name: resourcemanager
    ports:
      - 8088:8088
    environment:
      SERVICE_PRECONDITION: "namenode:9870 datanode1:9864"
    env_file:
      - ./hadoop.env
    networks:
      - hadoop

  # Hadoop slave 1
  datanode1:
    image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8
    container_name: datanode1
    volumes:
      - hadoop-datanode-1:/hadoop/dfs/data
    environment:
      SERVICE_PRECONDITION: "namenode:9870"
    env_file:
      - ./hadoop.env
    networks:
      - hadoop

  nodemanager1:
    image: bde2020/hadoop-nodemanager:2.0.0-hadoop3.2.1-java8
    container_name: nodemanager1
    volumes:
      - ./nodemanagers/entrypoint.sh:/entrypoint.sh
    environment:
      SERVICE_PRECONDITION: "namenode:9870 datanode1:9864 resourcemanager:8088"
    env_file:
      - ./hadoop.env
      - .env
    networks:
      - hadoop

  # Hadoop slave 2
  datanode2:
    image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8
    container_name: datanode2
    volumes:
      - hadoop-datanode-2:/hadoop/dfs/data
    environment:
      SERVICE_PRECONDITION: "namenode:9870"
    env_file:
      - ./hadoop.env
    networks:
      - hadoop

  nodemanager2:
    image: bde2020/hadoop-nodemanager:2.0.0-hadoop3.2.1-java8
    container_name: nodemanager2
    volumes:
      - ./nodemanagers/entrypoint.sh:/entrypoint.sh
    environment:
      SERVICE_PRECONDITION: "namenode:9870 datanode2:9864 resourcemanager:8088"
    env_file:
      - ./hadoop.env
      - .env
    networks:
      - hadoop

  historyserver:
    image: bde2020/hadoop-historyserver:2.0.0-hadoop3.2.1-java8
    container_name: historyserver
    ports:
      - 8188:8188
    environment:
      SERVICE_PRECONDITION: "namenode:9870 datanode1:9864 datanode2:9864 resourcemanager:8088"
    volumes:
      - hadoop-historyserver:/hadoop/yarn/timeline
    env_file:
      - ./hadoop.env
    networks:
      - hadoop
envsm3lx

envsm3lx1#

您将需要创建适合您计算机的定义大小的Docker卷,然后要求每个DN使用该卷。然后,当DN检查其卷的大小时,它应返回卷的大小而不是整个计算机的容量,并将其用作容量。

相关问题