如何检查应用程序id

v09wglhw  于 2021-05-27  发布在  Spark
关注(0)|答案(2)|浏览(580)

我试图运行bash脚本来运行spark submit和pyspark脚本,但没有成功。我想用“Yarn日志-applicationid”检查Yarn日志。我的问题是如何找到合适的应用程序id?
下面是我得到的错误的一些部分

sbtkgmzw

sbtkgmzw1#

1. Using Yarn Logs: 在日志中你可以看到 tracking URL: http://<nn>:8088/proxy/application_*****/ 如果复制并打开链接,您可以在resourcemanager中看到应用程序的所有日志。 2.Using Spark application: 从sparkcontext我们可以得到applicationid。

print(spark.sparkContext.aplicationId)
``` `3. Using yarn application command:` 使用 `yarn application --list` 命令获取集群上所有正在运行的应用程序,然后使用

yarn application --help
-appStates Works with -list to filter applications
based on input comma-separated list of
application states. The valid application
state can be one of the following:
ALL,NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUN
NING,FINISHED,FAILED,KILLED
-appTypes Works with -list to filter applications
based on input comma-separated list of
application types.
-help Displays help for all commands.
-kill Kills the application.
-list List applications. Supports optional use
of -appTypes to filter applications based
on application type, and -appStates to
filter applications based on application
state.
-movetoqueue Moves the application to a different
queue.
-queue Works with the movetoqueue command to
specify which queue to move an
application to.
-status Prints the status of the application.
`List all the finished applications:`
yarn application -appStates FINISHED -list

balp4ylt

balp4ylt2#

你也可以使用 curl 使用获取应用程序所需的详细信息 YARN Rest API .

state="RUNNING" // RUNNING, FAILED, COMPLETED.

user="" // userid from where you started job.

applicationTypes="spark" // Type of application

applicationName="<application_name>" // Your application name

url="http://<host_name>:8088/ws/v1/cluster/apps?state=${state}&user=${user}&applicationTypes=${applicationTypes}" // Build Rest API

applicationId=$(curl "${url}" | python -m json.tool | jq -r '.apps."app" | .[] | select(.name | contains('\"${applicationName}\"')) | .id')

输出

> echo $applicationId
application_1593019621736_42096

相关问题