我尝试在emr群集上使用mleap,但当我尝试使用它时,收到以下错误:
self._java_obj = _jvm().ml.combust.mleap.spark.SimpleSparkSerializer()
TypeError: 'JavaPackage' object is not callable
我从代码中加载jar文件
启动作业的脚本:
CLUSTER_ID=XXXXX
JOB_NAME=mleap_sample
SCRIPT_PATH=s3://XXXX/mleap_model.py
aws emr --profile xxx add-steps --cluster-id $CLUSTER_ID \
--steps Name=$JOB_NAME,Jar=command-runner.jar,\
Args=[spark-submit,--deploy-mode,client,\
--conf,spark.yarn.submit.waitAppCompletion=true,\
$SCRIPT_PATH],ActionOnFailure=CONTINUE
在我的代码里:
sc._jsc.addJar(
"s3a://xxxx/mleap_script/mleap-base_2.11-0.16.0-sources.jar")
sc._jsc.addJar(
"s3a://xxxx/mleap_script/mleap-core_2.11-0.16.0-sources.jar")
sc._jsc.addJar(
"s3a://xxxx/mleap_script/mleap-runtime_2.11-0.16.0-sources.jar")
sc._jsc.addJar(
"s3a://xxxx/mleap_script/mleap-spark-base_2.11-0.16.0-sources.jar")
sc._jsc.addJar(
"s3a://xxxx/mleap_script/mleap-spark-extension_2.11-0.16.0-sources.jar")
sc._jsc.addJar(
"s3a://xxxx/mleap_script/mleap-tensor_2.11-0.16.0-sources.jar")
在加载期间的emr日志中,我看到:
20/07/07 14:01:47 INFO SparkContext: Added JAR s3a://xxxx/mleap_script/mleap-base_2.11-0.16.0-sources.jar at s3a://mds-user-data-new/mleap_script/mleap-base_2.11-0.16.0-sources.jar with timestamp 1594130507784
20/07/07 14:01:47 INFO SparkContext: Added JAR s3a://xxxx/mleap_script/mleap-core_2.11-0.16.0-sources.jar at s3a://mds-user-data-new/mleap_script/mleap-core_2.11-0.16.0-sources.jar with timestamp 1594130507786
20/07/07 14:01:47 INFO SparkContext: Added JAR s3a://xxxx/mleap_script/mleap-runtime_2.11-0.16.0-sources.jar at s3a://mds-user-data-new/mleap_script/mleap-runtime_2.11-0.16.0-sources.jar with timestamp 1594130507788
20/07/07 14:01:47 INFO SparkContext: Added JAR s3a://xxxx/mleap_script/mleap-spark-base_2.11-0.16.0-sources.jar at s3a://mds-user-data-new/mleap_script/mleap-spark-base_2.11-0.16.0-sources.jar with timestamp 1594130507790
20/07/07 14:01:47 INFO SparkContext: Added JAR s3a://xxxx/mleap_script/mleap-spark-extension_2.11-0.16.0-sources.jar at s3a://mds-user-data-new/mleap_script/mleap-spark-extension_2.11-0.16.0-sources.jar with timestamp 1594130507793
20/07/07 14:01:47 INFO SparkContext: Added JAR s3a://xxxx/mleap_script/mleap-tensor_2.11-0.16.0-sources.jar at s3a://mds-user-data-new/mleap_script/mleap-tensor_2.11-0.16.0-sources.jar with timestamp 1594130507796
我用的是spark 2.4.5版
知道我为什么要面对这个问题吗?
ps:如果我在sagemaker笔记本上使用Pypark,我会收到相同的错误消息。
谢谢
暂无答案!
目前还没有任何答案,快来回答吧!