无法运行程序“-files”(在目录“.”中):错误=2,没有这样的文件或目录

uurity8g  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(455)

我正在尝试使用aws sdk ruby for amazon elasticmapreduce服务运行一个step,该服务使用hadoop,虽然我可以创建集群和step,但当使用web界面手动设置时,该步骤总是失败

emr = Aws::EMR::Client.new
    cluster_id = "*******"
    resp = emr.add_job_flow_steps({
      job_flow_id: cluster_id, # required
      steps: [ # required
        {
          name: "TestStep", # required
          action_on_failure: "CANCEL_AND_WAIT", # accepts TERMINATE_JOB_FLOW, TERMINATE_CLUSTER, CANCEL_AND_WAIT, CONTINUE
          hadoop_jar_step: { # required
            jar: 'command-runner.jar',
            args:[
                "-files",
                "s3://source123/mapper.py,s3://source123/source_reducer.py",
                "-mapper",
                "mapper.py",
                "-reducer",
                "source_reducer.py",
                "-input",
                "s3://source123/input/",
                "-output",
                "s3://source123/output/"
            ]
          },
        },
      ],
    })

我得到的错误是 Cannot run program "-files" (in directory "."): error=2, No such file or directory 有什么线索吗?

pw9qyyiw

pw9qyyiw1#

似乎添加hadoop流媒体的工作原理如下

emr = Aws::EMR::Client.new
    cluster_id = "*******"
    resp = emr.add_job_flow_steps({
      job_flow_id: cluster_id, # required
      steps: [ # required
        {
          name: "TestStep", # required
          action_on_failure: "CANCEL_AND_WAIT", # accepts TERMINATE_JOB_FLOW, TERMINATE_CLUSTER, CANCEL_AND_WAIT, CONTINUE
          hadoop_jar_step: { # required
            jar: 'command-runner.jar',
            args:[
                "hadoop-streaming",
                "-files",
                "s3://source123/mapper.py,s3://source123/source_reducer.py",
                "-mapper",
                "mapper.py",
                "-reducer",
                "source_reducer.py",
                "-input",
                "s3://source123/input/",
                "-output",
                "s3://source123/output/"
            ]
          },
        },
      ],
    })

相关问题