**结束。**此问题不符合堆栈溢出准则。它目前不接受答案。
**想改进这个问题吗?**更新问题,使其成为堆栈溢出的主题。
20天前关门了。
改进这个问题
在ubuntu18.04上,我安装了docker。在docker上我安装了一个ubuntu的映像。在这个ubuntu映像之上,我安装了spark3.1.1和hadoop2.7。
在spark上,我通过键入
启动主机.sh
当我尝试使用spark用户界面时
http://localhost:8080/
在我的本地ubuntu系统浏览器中,我得到了一条消息
无法连接
如何访问spark web用户界面?
以下是spark日志文件内容:
root@23d865d7f117:/opt/spark# cat logs/spark--org.apache.spark.deploy.master.Master-1-23d865d7f117.out
Spark Command: /usr/lib/jvm/java-11-openjdk-amd64/bin/java -cp /opt/spark/conf/:/opt/spark/jars/* -Xmx1g org.apache.spark.deploy.master.Master --host 23d865d7f117 --port 7077 --webui-port 8080
========================================
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/03/30 19:26:23 INFO Master: Started daemon with process name: 13951@23d865d7f117
21/03/30 19:26:23 INFO SignalUtils: Registering signal handler for TERM
21/03/30 19:26:23 INFO SignalUtils: Registering signal handler for HUP
21/03/30 19:26:23 INFO SignalUtils: Registering signal handler for INT
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark/jars/spark-unsafe_2.12-3.1.1.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
21/03/30 19:26:25 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/03/30 19:26:25 INFO SecurityManager: Changing view acls to: root
21/03/30 19:26:25 INFO SecurityManager: Changing modify acls to: root
21/03/30 19:26:25 INFO SecurityManager: Changing view acls groups to:
21/03/30 19:26:25 INFO SecurityManager: Changing modify acls groups to:
21/03/30 19:26:25 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
21/03/30 19:26:27 INFO Utils: Successfully started service 'sparkMaster' on port 7077.
21/03/30 19:26:27 INFO Master: Starting Spark master at spark://23d865d7f117:7077
21/03/30 19:26:27 INFO Master: Running Spark version 3.1.1
21/03/30 19:26:28 INFO Utils: Successfully started service 'MasterUI' on port 8080.
21/03/30 19:26:28 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://23d865d7f117:8080
21/03/30 19:26:28 INFO Master: I have been elected leader! New state: ALIVE
root@23d865d7f117:/opt/spark#
问题更新
按照你下面的答案,我试着应用你的答案,但遇到了一些困难。我做到了:
我已从容器中退出,并使用以下方式登录:
sudo docker运行-it-p8080:8080 ubuntu
然后我尝试启动独立的spark主服务器:
启动主机.sh
收到以下信息:
bash:start-master.sh:未找到命令
所以我又开始了设置环境变量的过程:
echo "export SPARK_HOME=/opt/spark" >> ~/.profile
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/bin:/sbin:$SPARK_HOME/bin:$SPARK_HOME/sbin
echo $PATH
echo "export PYSPARK_PYTHON=/usr/bin/python3" >> ~/.profile
以下是文件的输出。配置文件:
root@291b0eb654ea:/# cat ~/.profile
# ~/.profile: executed by Bourne-compatible login shells.
if [ "$BASH" ]; then
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
fi
mesg n 2> /dev/null || true
export SPARK_HOME=/opt/spark
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/bin:/sbin:/opt/spark/bin:/opt/spark/sbin
export PYSPARK_PYTHON=/usr/bin/python3
我还键入了以下命令:
root@291b0eb654ea:/#源~/.配置文件
我仍然收到以下信息:
bash:start-master.sh:未找到命令
我少了什么?
1条答案
按热度按时间uqjltbpv1#
正如评论中提到的,他正在运行容器
docker run -it ubuntu
没有端口Map。所以您应该运行以下命令:第一个在哪里
8080
是您要通过internet访问的端口,第二个是8080
是实际分配给容器中服务的端口号。让我举个例子来说明这一点:如你所知
http
端口号为80
,我有一个在80端口上运行的生产网站,还有一个在8080端口上运行的测试环境。对于测试环境,我的容器如下所示:
docker run -it -p8080:80 container
,所以容器上的实际端口是80,但我想将其Map到端口8080。