我使用的是Docker堆栈,它在同一台机器上实现了一个Hadoop Namenode、两个Datanodes、两个Node Manager、一个Resource Manager、一个History Server和其他技术。
我遇到了与HDFS UI中显示的HDFS配置容量相关的问题。
我使用的是一台容量为256GB的机器,并使用上面提到的两个数据节点实施。HDFS不是在两个节点之间分配总容量,而是通过为每个数据节点分配226.87GB来复制整个机器的容量。As you can see here
关于如何让HDFS显示正确的容量,您有什么想法吗?
下面是docker compose中实现上述hadoop技术的部分。
services:
# Hadoop master
namenode:
image: bde2020/hadoop-namenode:2.0.0-hadoop3.2.1-java8
container_name: namenode
ports:
- 9870:9870
- 8020:8020
volumes:
- ./namenode/home/${ADMIN_NAME:?err}:/home/${ADMIN_NAME:?err}
- ./namenode/hadoop-data:/hadoop-data
- ./namenode/entrypoint.sh:/entrypoint.sh
- hadoop-namenode:/hadoop/dfs/name
env_file:
- ./hadoop.env
- .env
networks:
- hadoop
resourcemanager:
restart: always
image: bde2020/hadoop-resourcemanager:2.0.0-hadoop3.2.1-java8
container_name: resourcemanager
ports:
- 8088:8088
environment:
SERVICE_PRECONDITION: "namenode:9870 datanode1:9864"
env_file:
- ./hadoop.env
networks:
- hadoop
# Hadoop slave 1
datanode1:
image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8
container_name: datanode1
volumes:
- hadoop-datanode-1:/hadoop/dfs/data
environment:
SERVICE_PRECONDITION: "namenode:9870"
env_file:
- ./hadoop.env
networks:
- hadoop
nodemanager1:
image: bde2020/hadoop-nodemanager:2.0.0-hadoop3.2.1-java8
container_name: nodemanager1
volumes:
- ./nodemanagers/entrypoint.sh:/entrypoint.sh
environment:
SERVICE_PRECONDITION: "namenode:9870 datanode1:9864 resourcemanager:8088"
env_file:
- ./hadoop.env
- .env
networks:
- hadoop
# Hadoop slave 2
datanode2:
image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8
container_name: datanode2
volumes:
- hadoop-datanode-2:/hadoop/dfs/data
environment:
SERVICE_PRECONDITION: "namenode:9870"
env_file:
- ./hadoop.env
networks:
- hadoop
nodemanager2:
image: bde2020/hadoop-nodemanager:2.0.0-hadoop3.2.1-java8
container_name: nodemanager2
volumes:
- ./nodemanagers/entrypoint.sh:/entrypoint.sh
environment:
SERVICE_PRECONDITION: "namenode:9870 datanode2:9864 resourcemanager:8088"
env_file:
- ./hadoop.env
- .env
networks:
- hadoop
historyserver:
image: bde2020/hadoop-historyserver:2.0.0-hadoop3.2.1-java8
container_name: historyserver
ports:
- 8188:8188
environment:
SERVICE_PRECONDITION: "namenode:9870 datanode1:9864 datanode2:9864 resourcemanager:8088"
volumes:
- hadoop-historyserver:/hadoop/yarn/timeline
env_file:
- ./hadoop.env
networks:
- hadoop
1条答案
按热度按时间envsm3lx1#
您将需要创建适合您计算机的定义大小的Docker卷,然后要求每个DN使用该卷。然后,当DN检查其卷的大小时,它应返回卷的大小而不是整个计算机的容量,并将其用作容量。