我已经编写了我的第一个map reduce程序,当我在eclipse中运行它时,它会写入输出文件并按预期工作。但是,当我使用hadoop jar myjar.jar从命令行运行它时,结果不会写入输出文件。正在创建输出文件(\u success和part-r-0000),但它们为空。是否存在持久性问题?。reduce input records=12,reduce output records=0。但是如果我在eclipse中这样做的话,这个值不是零。在eclipse中,reduce输出记录不是0。感谢您的帮助。谢谢
[cloudera@quickstart Desktop]$ sudo hadoop jar checkjar.jar hdfs://quickstart.cloudera:8020/user/cloudera/input.csv hdfs://quickstart.cloudera:8020/user/cloudera/output9
15/04/28 22:09:06 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
15/04/28 22:09:07 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
15/04/28 22:09:08 INFO input.FileInputFormat: Total input paths to process : 1
15/04/28 22:09:09 INFO mapreduce.JobSubmitter: number of splits:1
15/04/28 22:09:09 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1430279123629_0011
15/04/28 22:09:10 INFO impl.YarnClientImpl: Submitted application application_1430279123629_0011
15/04/28 22:09:10 INFO mapreduce.Job: The url to track the job: http://quickstart.cloudera:8088/proxy/application_1430279123629_0011/
15/04/28 22:09:10 INFO mapreduce.Job: Running job: job_1430279123629_0011
15/04/28 22:09:22 INFO mapreduce.Job: Job job_1430279123629_0011 running in uber mode : false
15/04/28 22:09:22 INFO mapreduce.Job: map 0% reduce 0%
15/04/28 22:09:32 INFO mapreduce.Job: map 100% reduce 0%
15/04/28 22:09:46 INFO mapreduce.Job: map 100% reduce 100%
15/04/28 22:09:46 INFO mapreduce.Job: Job job_1430279123629_0011 completed successfully
15/04/28 22:09:46 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=265
FILE: Number of bytes written=211403
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=365
HDFS: Number of bytes written=0
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=8175
Total time spent by all reduces in occupied slots (ms)=10124
Total time spent by all map tasks (ms)=8175
Total time spent by all reduce tasks (ms)=10124
Total vcore-seconds taken by all map tasks=8175
Total vcore-seconds taken by all reduce tasks=10124
Total megabyte-seconds taken by all map tasks=8371200
Total megabyte-seconds taken by all reduce tasks=10366976
Map-Reduce Framework
Map input records=12
Map output records=12
Map output bytes=235
Map output materialized bytes=265
Input split bytes=120
Combine input records=0
Combine output records=0
Reduce input groups=2
Reduce shuffle bytes=265
Reduce input records=12
Reduce output records=0
Spilled Records=24
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=172
CPU time spent (ms)=1150
Physical memory (bytes) snapshot=346574848
Virtual memory (bytes) snapshot=1705988096
Total committed heap usage (bytes)=196481024
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=245
File Output Format Counters
Bytes Written=0
减速器.java
package com.mapreduce.assgn4;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;
public class JoinReducer
extends Reducer<Text, Text, Text, Text> {
@Override
public void reduce(Text key, Iterable<Text> values,
Context context)
throws IOException, InterruptedException {
List<String> tableoneTuples = new ArrayList<String>();
List<String> tabletwoTuples = new ArrayList<String>();
for (Text value : values) {
String[] splitValues = value.toString().split("#");
String tableName = splitValues[0];
if(tableName.equals(JoinMapper.tableone))
{
tableoneTuples.add(splitValues[1]);
}
else
{
tabletwoTuples.add(splitValues[1]);
}
}
System.out.println(tableoneTuples.size());
System.out.println(tabletwoTuples.size());
String FinaljoinString = null;
for(String tableoneValue: tableoneTuples)
{
for (String tabletwoValue: tabletwoTuples)
{
FinaljoinString = tableoneValue+","+tabletwoValue;
FinaljoinString = key.toString()+","+FinaljoinString;
context.write(null, new Text(FinaljoinString));
}
}
}
}
1条答案
按热度按时间qltillow1#
你的context.write-in-reducer有错误。你需要有nullwriteable才能在输出中有null,