我正试图在googlecomputeengine上安装hadoop集群，我一直在遵循这些说明。一切似乎都很顺利，直到我跑了：
./compute_cluster_for_hadoop.py setup <project ID> <bucket name>使用我创建的项目id和bucket名称。脚本似乎无法访问某些内容，并与403崩溃；下面是带有错误消息的输出的结尾：
Uploading ...kages/ca-certificates-java_20121112+nmu2_all.deb: 14.57 KB/14.57 KB
Uploading ...duce/tmp/deb_packages/libnspr4_4.9.2-1_amd64.deb: 316 B/316 B
Uploading ...e/tmp/deb_packages/libnss3-1d_3.14.3-1_amd64.deb: 318 B/318 B
Uploading ...dk-6-jre-headless_6b27-1.12.6-1~deb7u1_amd64.deb: 366 B/366 B
Uploading ...duce/tmp/deb_packages/libnss3_3.14.3-1_amd64.deb: 315 B/315 B
ResumableUploadAbortException: 403 Forbidden
AccessDeniedException: 403 Forbidden
AccessDeniedException: 403 Forbidden
AccessDeniedException: 403 Forbidden
AccessDeniedException: 403 Forbidden
ResumableUploadAbortException: 403 Forbidden
AccessDeniedException: 403 Forbidden
CommandException: 7 files/objects could not be transferred.
########## ERROR ##########
Failed to copy Hadoop and Java packages to Cloud Storage gs://<bucket name>/mapreduce/tmp/
###########################
Traceback (most recent call last):
File "./compute_cluster_for_hadoop.py", line 230, in <module>
main()
File "./compute_cluster_for_hadoop.py", line 226, in main
ComputeClusterForHadoop().ParseArgumentsAndExecute(sys.argv[1:])
File "./compute_cluster_for_hadoop.py", line 222, in ParseArgumentsAndExecute
params.handler(params)
File "./compute_cluster_for_hadoop.py", line 36, in SetUp
gce_cluster.GceCluster(flags).EnvironmentSetUp()
File "/Path/To/solutions-google-compute-engine-cluster-for-hadoop/gce_cluster.py", line 149, in EnvironmentSetUp
raise EnvironmentSetUpError('Environment set up failed.')
gce_cluster.EnvironmentSetUpError: Environment set up failed.
1条答案
按热度按时间fxnxkyjh1#
我建议您改用google提供的更新、实时和维护更多的“bdutil”包。您可以在gcp hadoop公告论坛中找到详细信息。
如果您转到最新的公告，您将找到最新的“bdutil”包（当前为0.36.4）的链接。它将简化集群部署，并支持hadoop和spark集群。
另外，我想推荐：
在gce的机器中部署集群。它将使这一过程更加迅速和可靠。
在文件中
bdutil_env.sh，更改参数GCUTIL_SLEEP_TIME_BETWEEN_ASYNC_CALLS_SECONDS从0.1到0.5（对我来说，它修复了重复出现的部署错误）