generic选项-d在hadoop 0.20.2中不支持吗?

ldfqzlk8  于 2021-06-02  发布在  Hadoop
关注(0)|答案(2)|浏览(328)

我正在尝试使用-d generic选项从控制台设置configuration属性。
这是我的控制台输入:

$ hadoop jar hadoop-0.20.2/gtee.jar dd.MaxTemperature -D file.pattern=2007.* /inputdata /outputdata

但我在密码上做了交叉验证

Configuration conf;
System.out.println(conf.get("file.pattern"));

结果为空输出。这里会有什么问题,为什么属性“file.pattern”的值不显示?谁能帮帮我吗。
谢谢
编辑的部分:
驱动程序代码:

public int run(String[] args) throws Exception {
    Job job = new Job();
    job.setJarByClass(MaxTemperature.class);
    job.setJobName("MaxTemperature");
    Configuration conf = new Configuration();
    FileSystem fs = FileSystem.get(URI.create(args[0]), conf);
//  System.out.println(conf.get("file.pattern"));

    if (fs.exists(new Path(args[1]))) {
        fs.delete(new Path(args[1]), true);
    }

    System.out.println(conf.get("file.pattern"));

    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);

    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileInputFormat.setInputPathFilter(job, RegexExcludePathFilter.class);

    FileOutputFormat.setOutputPath(job, new Path(args[1]));
    job.setOutputFormatClass(TextOutputFormat.class);

    job.setMapperClass(MapMapper.class);
    job.setCombinerClass(Mapreducers.class);
    job.setReducerClass(Mapreducers.class);

    return job.waitForCompletion(true) ? 0 : 1;
}

public static void main(String[] args) throws Exception {
    int xx = ToolRunner.run(new Configuration(),new MaxTemperature(), args);
    System.exit(xx);
}

路径筛选器实现:

public static class RegexExcludePathFilter extends Configured implements
        PathFilter {
        //String pattern = "2007.[0-1]?[0-2].[0-9][0-9].txt" ;
        Configuration conf; 
        Pattern pattern;

    @Override
    public boolean accept(Path path) {
        Matcher m = pattern.matcher(path.toString());
        return m.matches();
    }

    @Override
    public void setConf(Configuration conf) {
        this.conf = conf;
        pattern = Pattern.compile(conf.get("file.pattern"));
        System.out.println(pattern);
    }

}
i1icjdpr

i1icjdpr1#

为了确认在版本20.2中支持you-d选项,但是这需要您实现从命令行读取变量的工具接口

Configuration conf = new Configuration(); //this is the issue
  // When implementing tool use this
   Configuration conf = this.getConf();
gfttwv5a

gfttwv5a2#

你在传递它的时候中间有一个空格,这不是你应该做的。相反,请尝试:

-Dfile.pattern=2007.*

相关问题