hadoop context.write()输出打印带有文本对象的nan

1tuwyuhd  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(492)

这是我第一次使用hadoop,我在写入输出文件时遇到了问题。当我用system.out打印值时,它显示得很好,但是使用context.write(key,value)将值打印为nan。
例子:

  1. System.out.println(stockName.toString() + " " + result.toString());

正确输出到用户日志:

  1. AAPL.csv 0.076543

但使用:

  1. context.write(stockName, result);

输出:

  1. AAPL.csv NaN

result和stockname都是以前设置的text()对象。
我还包括了我的整个reduce函数。任何想法都会很好,因为我已经尝试了我能想到的一切,谢谢!

  1. public static class Reduce extends Reducer<Text, Text, Text, Text> {
  2. private Text stockName = new Text();
  3. private ArrayList<Float> monthlyReturn = new ArrayList<Float>();
  4. private String previousMonth = "";
  5. private float numOfMonths = 0;
  6. private float startPrice = 0;
  7. private float endPrice = 0;
  8. private Text result = new Text();
  9. public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
  10. // Set the Stock Name as the Key
  11. stockName.set(key);
  12. for (Text val: values) {
  13. System.out.println(val);
  14. // Parse date & adjusted close
  15. String[] stockValues = val.toString().split(",");
  16. if (stockValues.length < 2) {
  17. continue;
  18. }
  19. String month = stockValues[0];
  20. String priceInput = stockValues[1];
  21. float closingPrice = Float.parseFloat(priceInput);
  22. // First time around setup.
  23. if (startPrice == 0 && previousMonth.equals("")) {
  24. startPrice = closingPrice;
  25. previousMonth = month;
  26. }
  27. /*
  28. * We check if the month has changed, and that we're not just starting.
  29. * If the month changed, increment the number of months we have seen, and run a calculation
  30. * for monthly return.
  31. *
  32. * closePrice is set to every stock value. The startPrice is only set when the month changes.
  33. * When the month does change, we take the last set closePrice to run our calculation, and
  34. * then set the new startPrice.
  35. */
  36. if (!month.equals(previousMonth) && endPrice != 0) {
  37. numOfMonths += 1;
  38. monthlyReturn.add((endPrice - startPrice)/startPrice);
  39. startPrice = closingPrice;
  40. }
  41. previousMonth = month;
  42. endPrice = closingPrice;
  43. }
  44. // Add on the last month value
  45. numOfMonths += 1;
  46. monthlyReturn.add((endPrice - startPrice)/startPrice);
  47. /*
  48. * Generate the volatility. The equation is as follows:
  49. *
  50. * 1. xbar = sum(xi)/numOfMonth -> sum is over all values from 0 to N in monthlyReturn
  51. * 2. xsum = sum( (xi-xbar)^2 ) from 0 to N in monthlyReturn
  52. * 3. volatility = sqrt( (1/numOfMonth-1)*xsum )
  53. */
  54. // 1.
  55. float xiSum = 0;
  56. for (int i =0; i<monthlyReturn.size(); i++) {
  57. xiSum += monthlyReturn.get(i);
  58. }
  59. float xBar = xiSum/numOfMonths;
  60. // 2.
  61. double xSum = 0;
  62. for (int i=0; i<monthlyReturn.size(); i++) {
  63. xSum += Math.pow(monthlyReturn.get(i) - xBar, 2);
  64. }
  65. // 3.
  66. double root = (1/(numOfMonths-1))*xSum;
  67. result.set(String.valueOf(Math.sqrt(root)));
  68. System.out.println(stockName.toString() + " " + result.toString());
  69. context.write(stockName, result);
  70. }
  71. }
  72. public static void main(String[] args) throws Exception {
  73. Job job = Job.getInstance();
  74. job.setJarByClass(StockVolatility.class);
  75. job.setMapperClass(Map.class);
  76. job.setCombinerClass(Reduce.class);
  77. job.setReducerClass(Reduce.class);
  78. job.setOutputKeyClass(Text.class);
  79. job.setOutputValueClass(Text.class);
  80. job.setInputFormatClass(TextInputFormat.class);
  81. job.setOutputFormatClass(TextOutputFormat.class);
  82. FileInputFormat.addInputPath(job, new Path(args[0]));
  83. FileOutputFormat.setOutputPath(job, new Path(args[1]));
  84. job.waitForCompletion(true);
  85. }
qacovj5a

qacovj5a1#

不要使用job.setcombinerclass(reduce.class);我做了那件事之后,我的问题就解决了。

相关问题