远程运行hadoop作业_大数据知识库

我正在尝试从群集中外部运行mapreduce作业。
e、 g.hadoop集群在linux机器上运行。我们有一个在windows机器上运行的web应用程序。我们想从这个远程web应用程序运行hadoop作业。我们希望检索hadoop输出目录并将其显示为一个图形。
我们编写了以下代码：

Configuration conf = new Configuration();
Job job = new Job(conf);
conf.set("mapred.job.tracker", "192.168.56.101:54311"); 
conf.set("fs.default.name", "hdfs://192.168.56.101:54310");
job.setJarByClass(Analysis.class) ;
//job.setOutputKeyClass(Text.class);
//job.setOutputValueClass(IntWritable.class);
job.setMapperClass(Map.class);
job.setReducerClass(Reduce.class);
//job.set
job.setInputFormatClass(CustomFileInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.waitForCompletion(true);

这就是我们得到的错误。即使我们关闭了hadoop1.1.2集群，错误仍然是一样的。

14/03/07 00:23:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/03/07 00:23:37 ERROR security.UserGroupInformation: PriviledgedActionException as:user cause:java.io.IOException: Failed to set permissions of path: \tmp\hadoop-user\mapred\staging\user818037780\.staging to 0700
Exception in thread "main" java.io.IOException: Failed to set permissions of path: \tmp\hadoop-user\mapred\staging\user818037780\.staging to 0700
at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:691)
at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:664)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:193)
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
at LineCounter.main(LineCounter.java:86)

从远程系统运行时，应以远程用户身份运行。你可以在你的主课上做如下：

public static void main(String a[]) {
     UserGroupInformation ugi
     = UserGroupInformation.createRemoteUser("root");
     try {
        ugi.doAs(new PrivilegedExceptionAction<Void>() {
            public Void run() throws Exception {
                 Configuration conf = new Configuration();
                 Job job = new Job(conf);
                 conf.set("hadoop.job.ugi", "root");
                 // write your remaining piece of code here. 
                return null;
            }
        });
    } catch (Exception e) {
        e.printStackTrace();
    }
}

另外，在提交mapreduce作业时，它应该将java类及其相关JAR复制到hadoop集群，在那里执行mapreduce作业。
因此，您需要用所有依赖的jar文件inits manifest classpath为代码创建一个可运行的jar（在本例中使用主类分析）。然后使用命令行运行jar文件

java -jar job-jar-with-dependencies.jar arguments

嗯！

展开查看全部

远程运行hadoop作业

1条答案

相关问题

热门标签

最新问答