空指针异常-hadoop mapreduce作业

mznpcxlj  于 2021-06-02  发布在  Hadoop
关注(0)|答案(2)|浏览(456)

我是hadoop和java的初学者,我正在编写map,reduce函数,将一组纬度和经度基于接近度聚集在一起,并设置一个震级(一个簇中的lat,long pair的数量)和一个代表lat,long pair(到目前为止,这是遇到的第一个lat,long pair)
这是我的密码:

  1. package org.myorg;
  2. import java.io.IOException;
  3. import java.util.*;
  4. import org.apache.hadoop.fs.Path;
  5. import org.apache.hadoop.conf.*;
  6. import org.apache.hadoop.io.*;
  7. import org.apache.hadoop.mapreduce.*;
  8. import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
  9. import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
  10. import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
  11. import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
  12. import util.hashing.*;
  13. public class LatLong {
  14. public static class Map extends Mapper<Object, Text, Text, Text> {
  15. //private final static IntWritable one = new IntWritable(1);
  16. public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
  17. String line = value.toString();
  18. String[] longLatArray = line.split(",");
  19. double longi = Double.parseDouble(longLatArray[0]);
  20. double lat = Double.parseDouble(longLatArray[1]);
  21. //List<Double> origLatLong = new ArrayList<Double>(2);
  22. //origLatLong.add(lat);
  23. //origLatLong.add(longi);
  24. Geohash inst = Geohash.getInstance();
  25. //encode is the library's encoding function
  26. String hash = inst.encode(lat,longi);
  27. //Using the first 5 characters just for testing purposes
  28. //Need to find the right one later
  29. int accuracy = 4;
  30. //hash of the thing is shortened to whatever I figure out
  31. //to be the right size of each tile
  32. Text shortenedHash = new Text(hash.substring(0,accuracy));
  33. Text origHash = new Text(hash);
  34. context.write(shortenedHash, origHash);
  35. }
  36. }
  37. public static class Reduce extends Reducer<Text, Text, Text, Text> {
  38. private IntWritable totalTileElementCount = new IntWritable();
  39. private Text latlongimag = new Text();
  40. private Text dataSeparator = new Text();
  41. @Override
  42. public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
  43. int elementCount = 0;
  44. boolean first = true;
  45. Iterator<Text> it= values.iterator();
  46. String lat = new String();
  47. String longi = new String();
  48. Geohash inst = Geohash.getInstance();
  49. while (it.hasNext()) {
  50. elementCount = elementCount+1;
  51. if(first)
  52. {
  53. lat = Double.toString((inst.decode(it.toString()))[0]);
  54. longi = Double.toString((inst.decode(it.toString()))[1]);
  55. first = false;
  56. }
  57. @SuppressWarnings("unused")
  58. String blah = it.next().toString();
  59. }
  60. totalTileElementCount.set(elementCount);
  61. //Geohash inst = Geohash.getInstance();
  62. String mag = totalTileElementCount.toString();
  63. latlongimag.set(lat+","+ longi +","+mag+",");
  64. dataSeparator.set("");
  65. context.write(latlongimag, dataSeparator );
  66. }
  67. }
  68. public static void main(String[] args) throws Exception {
  69. Configuration conf = new Configuration();
  70. Job job = new Job(conf, "wordcount");
  71. job.setJarByClass(LatLong.class);
  72. job.setOutputKeyClass(Text.class);
  73. job.setOutputValueClass(Text.class);
  74. job.setMapperClass(Map.class);
  75. job.setReducerClass(Reduce.class);
  76. job.setInputFormatClass(TextInputFormat.class);
  77. job.setOutputFormatClass(TextOutputFormat.class);
  78. FileInputFormat.addInputPath(job, new Path(args[0]));
  79. FileOutputFormat.setOutputPath(job, new Path(args[1]));
  80. job.waitForCompletion(true);
  81. }
  82. }

我要参加npe。我不知道如何测试这个,我也无法在我的代码中找到错误。
hadoop错误:

  1. java.lang.NullPointerException
  2. at util.hashing.Geohash.decode(Geohash.java:41)
  3. at org.myorg.LatLong$Reduce.reduce(LatLong.java:67)
  4. at org.myorg.LatLong$Reduce.reduce(LatLong.java:1)
  5. at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
  6. at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:663)
  7. at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:426)
  8. at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
  9. at java.security.AccessController.doPrivileged(Native Method)
  10. at javax.security.auth.Subject.doAs(Subject.java:396)
  11. at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
  12. at org.apache.hadoop.mapred.Child.main(Child.java:249)

geohash库中的decode函数返回一个双精度数组。任何指点都将不胜感激!谢谢你的时间!
edit1(测试后):
我意识到问题在于reduce函数中需要有it.next().tostring(),而不仅仅是it.tostring(),但是当我做了这个更改并进行测试时,我发现了这个错误,我不知道为什么在while循环条件中检查hasnext()时会出现这个错误。

  1. java.util.NoSuchElementException: iterate past last value
  2. at org.apache.hadoop.mapreduce.ReduceContext$ValueIterator.next(ReduceContext.java:159)
  3. at org.myorg.LatLong$Reduce.reduce(LatLong.java:69)
  4. at org.myorg.LatLong$Reduce.reduce(LatLong.java:1)
  5. at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
  6. at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:663)
  7. at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:426)
  8. at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
  9. at java.security.AccessController.doPrivileged(Native Method)
  10. at javax.security.auth.Subject.doAs(Subject.java:396)
  11. at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
  12. at org.apache.hadoop.mapred.Child.main(Child.java:249)

edit2(进一步测试):解决方案
我不止一次地调用它.next(),作为一个迭代器,这只会导致它继续,两次,在最后一次迭代中,它检查条件并输入,但我随后调用它.next()两次,这会导致问题,因为只有一个下一个元素(最后一个)

kxxlusnw

kxxlusnw1#

你还打电话吗 toStringit ,而不是 it.next() ,所以你应该改变

  1. lat = Double.toString((inst.decode(it.toString()))[0]);
  2. longi = Double.toString((inst.decode(it.toString()))[1]);

进入之内

  1. String cords = it.next().toString();
  2. lat = Double.toString((inst.decode(cords))[0]);
  3. longi = Double.toString((inst.decode(cords))[1]);

别这样 inst.decode(it.next().toString()) 因为它会打电话来 it.next() 一次两次 while 迭代。
之后不要打电话 String blah = it.next().toString(); 因为你会得到 java.util.NoSuchElementException: iterate past last value ,原因同上。
当你移除 String blah = it.next().toString(); 记住,万一 first = false 你永远进不去 if(first) 永远不要打电话 String cords = it.next().toString(); 所以呢 it.hasNext() 总会回来的 true 你永远不会离开 while 循环,所以添加适当的条件语句。

uemypmqf

uemypmqf2#

这意味着要么你的“it”是空的,要么你解码后得到空的。为它们设置空检查。

相关问题