dataproc忽略spark配置

vuv7lop3 于 2021-05-16 发布在 Spark

关注(0)|答案(2)|浏览(658)

我正在dataproc集群中运行一个below spark submit命令，但是我注意到很少有spark配置被忽略。我能知道他们被忽视的原因吗？

gcloud dataproc jobs submit spark --cluster=<Cluster> --class=<class_name> --jars=<list_of_jars> --region=<region> --files=<list_of_files> --properties=spark.driver.extraJavaOptions="-Dconfig.file=application_dev.json -Dlog4j.configuration=log4j.properties",spark.executor.extraJavaOptions="-Dconfig.file=application_dev.json -Dlog4j.configuration=log4j.properties, spark.executor.instances=36, spark.executor.cores=4, spark.executor.memory=4G, spark.driver.memory=8G, spark.shuffle.service.enabled=true, spark.yarn.maxAppAttempts=1, spark.sql.shuffle.partitions=200, spark.executor.memoryOverhead=7680, spark.driver.maxResultSize=0, spark.port.maxRetries=250, spark.dynamicAllocation.initialExecutors=20, spark.dynamicAllocation.minExecutors=10"

Warning: Ignoring non-Spark config property:  spark.driver.maxResultSize
Warning: Ignoring non-Spark config property:  spark.driver.memory
Warning: Ignoring non-Spark config property:  spark.dynamicAllocation.minExecutors
Warning: Ignoring non-Spark config property:  spark.executor.cores
Warning: Ignoring non-Spark config property:  spark.port.maxRetries
Warning: Ignoring non-Spark config property:  spark.yarn.maxAppAttempts
Warning: Ignoring non-Spark config property:  spark.dynamicAllocation.initialExecutors
Warning: Ignoring non-Spark config property:  spark.executor.memory
Warning: Ignoring non-Spark config property:  spark.executor.memoryOverhead
Warning: Ignoring non-Spark config property:  spark.sql.shuffle.partitions
Warning: Ignoring non-Spark config property:  spark.executor.instances

apache-spark google-cloud-platform google-cloud-dataproc

来源：https://stackoverflow.com/questions/65229816/dataproc-ignoring-spark-configuration

2条答案

按热度按时间

njthzxwz1#

试试下面的。他们不是 extraJavaOptions ，但属于 properties .

gcloud dataproc jobs submit spark --cluster=<Cluster> --class=<class_name> --jars=<list_of_jars> --region=<region> --files=<list_of_files> --properties=spark.driver.extraJavaOptions="-Dconfig.file=application_dev.json -Dlog4j.configuration=log4j.properties",spark.executor.extraJavaOptions="-Dconfig.file=application_dev.json -Dlog4j.configuration=log4j.properties",spark.executor.instances=36,spark.executor.cores=4,spark.executor.memory=4G,spark.driver.memory=8G,spark.shuffle.service.enabled=true,spark.yarn.maxAppAttempts=1,spark.sql.shuffle.partitions=200,spark.executor.memoryOverhead=7680,spark.driver.maxResultSize=0,spark.port.maxRetries=250,spark.dynamicAllocation.initialExecutors=20,spark.dynamicAllocation.minExecutors=10

以更易读的形式：

gcloud dataproc jobs submit spark --cluster=<Cluster> --class=<class_name> --jars=<list_of_jars> --region=<region> --files=<list_of_files> 
--properties=spark.driver.extraJavaOptions="
    -Dconfig.file=application_dev.json
    -Dlog4j.configuration=log4j.properties
",spark.executor.extraJavaOptions="
    -Dconfig.file=application_dev.json
    -Dlog4j.configuration=log4j.properties
",
spark.executor.instances=36,
spark.executor.cores=4,
spark.executor.memory=4G,
spark.driver.memory=8G,
spark.shuffle.service.enabled=true,
spark.yarn.maxAppAttempts=1,
spark.sql.shuffle.partitions=200,
spark.executor.memoryOverhead=7680,
spark.driver.maxResultSize=0,
spark.port.maxRetries=250,
spark.dynamicAllocation.initialExecutors=20,
spark.dynamicAllocation.minExecutors=10

赞(0）回复(0）举报 2021-05-17

mitkmikd2#

你能试试这个吗？

gcloud dataproc jobs submit spark \
  --cluster=<Cluster> \
  --class=<class_name> \
  --jars=<list_of_jars> \
  --region=<region> \
  --files=<list_of_files> \
  --properties=^#^spark.driver.extraJavaOptions="-Dconfig.file=application_dev.json -Dlog4j.configuration=log4j.properties"#spark.executor.extraJavaOptions="-Dconfig.file=application_dev.json -Dlog4j.configuration=log4j.properties"#spark.executor.instances=36#spark.executor.cores=4#spark.executor.memory=4G#spark.driver.memory=8G#spark.shuffle.service.enabled=true#spark.yarn.maxAppAttempts=1#spark.sql.shuffle.partitions=200#spark.executor.memoryOverhead=7680#spark.driver.maxResultSize=0#spark.port.maxRetries=250#spark.dynamicAllocation.initialExecutors=20#spark.dynamicAllocation.minExecutors=10

赞(0）回复(0）举报 2021-05-16

我来回答

dataproc忽略spark配置

2条答案

相关问题

热门标签

最新问答