我已经安装了Spark 3.3.1
,并且它以前是用spark-shell
和pyspark
命令运行的。但是在我安装了Hadoop 3.3.1
之后,似乎pyspark
命令不能正常工作,这是运行该命令的结果:
C:\Users\A>pyspark2 --num-executors 4 --executor-memory 1g
[I 2022-11-20 22:36:09.100 LabApp] JupyterLab extension loaded from C:\Users\A\AppData\Local\Programs\Python\Python311\Lib\site-packages\jupyterlab
[I 2022-11-20 22:36:09.100 LabApp] JupyterLab application directory is C:\Users\A\AppData\Local\Programs\Python\Python311\share\jupyter\lab
[I 22:36:09.107 NotebookApp] Serving notebooks from local directory: C:\Users\A
[I 22:36:09.107 NotebookApp] Jupyter Notebook 6.5.2 is running at:
[I 22:36:09.107 NotebookApp] http://localhost:8888/?token=0fca9f0378976c7af19886970c9e801ac27a8d1a209528db
[I 22:36:09.108 NotebookApp] or http://127.0.0.1:8888/?token=0fca9f0378976c7af19886970c9e801ac27a8d1a209528db
[I 22:36:09.108 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 22:36:09.189 NotebookApp]
To access the notebook, open this file in a browser:
file:///C:/Users/A/AppData/Roaming/jupyter/runtime/nbserver-8328-open.html
Or copy and paste one of these URLs:
http://localhost:8888/?token=0fca9f0378976c7af19886970c9e801ac27a8d1a209528db
or http://127.0.0.1:8888/?token=0fca9f0378976c7af19886970c9e801ac27a8d1a209528db
0.01s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
它打开了Jupyter notebook
,但是Spark标志没有显示,Python shell也不能像以前那样在CMD
中使用。但是spark-shell
仍然可以如下工作:
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://168.150.8.52:4040
Spark context available as 'sc' (master = local[*], app id = local-1669062477403).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.3.1
/_/
Using Scala version 2.12.15 (OpenJDK 64-Bit Server VM, Java 11.0.16.1)
Type in expressions to have them evaluated.
Type :help for more information.
scala> 22/11/21 12:28:12 WARN ProcfsMetricsGetter: Exception when trying to compute pagesize, as a result reporting of ProcessTree metrics is stopped
scala>
2条答案
按热度按时间1rhkuytd1#
您的路径已更改为使用sparks python发行版。您可以了解更多关于此here的信息。
尝试:
echo $PATH
然后看看你有多少条Python,我打赌你不止一条。
von4xj4u2#
它打开Jupyter笔记本,但没有显示Spark徽标,Python shell也不可用
Jupyter是一个Python shell(默认情况下)。
Spark没有附带
pyspark2
命令,所以看起来你已经对你的环境做了一些自定义。而且,如果你设置了一个特定的环境变量,它只会默认打开Jupyter。徽标不一定能告诉您它正在工作。请尝试创建会话