mapreduce传递命令行参数

5lwkijsr 于 2021-06-03 发布在 Hadoop

关注(0)|答案(3)|浏览(556)

我正在尝试使用新的api进行map reduce，并将一个常规的expression作为a-d命令行参数传入，但是没有得到它。结果是pattern.compile（pattern）得到一个 NullPointerException 我的Map器代码是；

public class MdacMapper extends Mapper<Text, Text, Text, Text> {

    private Pattern compiledPattern;

    public void setup(Context context) {
        Configuration config = context.getConfiguration();
        String pattern = config.get("mapper.pattern");
        compiledPattern = Pattern.compile(pattern);
    }

    // mapper
}

我的控制器代码是；

public class JobController {

    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();

       Job job = new Job(conf);
       job.setJarByClass(JobController.class);
       job.setInputFormatClass(TextInputFormat.class);
       job.setOutputFormatClass(TextOutputFormat.class);

       FileInputFormat.setInputPaths(job, new Path(args[0]));
       FileOutputFormat.setOutputPath(job, new Path(args[1]));

       job.setMapperClass(MdacMapper.class);
       job.setReducerClass(MdacReducer.class);
       job.setMapOutputKeyClass(Text.class);
       job.setMapOutputValueClass(Text.class);
       job.setOutputKeyClass(Text.class);
       job.setOutputValueClass(Text.class);

       job.waitForCompletion(true);
       job.submit();
    }

}
最后，我要做的命令行调用是；

hadoop jar mapreducer.jar /input/aclogdrop /output/acjava \ 
 -Dmapper.pattern=".*\|(\d{8}):\d*\.\d*\|(\d+)\|AC_ADO_QRY\(\d*?\)\|\d?\s?\[\s?(.*?)\]"

有什么建议为什么我不能选择配置参数mapper.pattern吗？

hadoop mapreduce hadoop2

来源：https://stackoverflow.com/questions/23435155/mapreduce-passing-command-line-parameters

3条答案

按热度按时间

0aydgbwb1#

此外，使用工具接口时，必须首先在命令中指定泛型选项（例如-d property=value），然后指定应用程序参数。
正确的命令如下所示：

hadoop jar mapreducer.jar -D mapper.pattern=".*\|(\d{8}):\d*\.\d*\|(\d+)\|AC_ADO_QRY\(\d*?\)\|\d?\s?\[\s?(.*?)\]" /input/aclogdrop /output/acjava

赞(0）回复(0）举报 2021-06-03

svmlkihl2#

也许你应该用 GenericOptionsParser .
下面是api GenericOptions解析器

赞(0）回复(0）举报 2021-06-03

dy1byipe3#

您想使用工具界面-例如：

public class WordCount extends Configured implements Tool {

  public static class Map
      extends Mapper<Object, Text, Text, IntWritable> {

    private final static IntWritable one = new IntWritable(1);
    private final static Text word = new Text();

    public void setup(Context context) {
        Configuration config = context.getConfiguration();
        String wordstring = config.get("mapper.word");
        word.set(wordstring);
    }

    /* wordcount mapper, reducer */
  }

  public static void main(String[] args) throws Exception {
    int res = ToolRunner.run(new Configuration(), new WordCount(), args);
    System.exit(res);
  }

  public int run(String[] args) throws Exception {

    if (args.length != 2) {
      System.err.println("Usage: wordcount <in> <out>");
      System.exit(2);
    }

    Configuration conf = this.getConf();

    Job job = new Job(conf, "WordCount");
    job.setJarByClass(WordCount.class);

    job.setMapperClass(Map.class);
    job.setReducerClass(Reduce.class);

    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);

    FileInputFormat.setInputPaths(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));

    return job.waitForCompletion(true) ? 0 : 1;
  }
}

赞(0）回复(0）举报 2021-06-03

我来回答

mapreduce传递命令行参数

3条答案

相关问题

热门标签

最新问答