我有一个简单的螺栓,从Kafka喷口读取数据,然后将数据写入hdfs目录。问题是,在集群停止之前,bolt不会写入。我如何确保当bolt从kafka喷口读取一个元组时,它会立即将其写入hdfs,或者至少写入每个“n”个条目(我正在使用cdh 4.4、hadoop 2.0)
螺栓的 java 语:
public class PrinterBolt10 extends BaseRichBolt{
private OutputCollector collector;
private String values;
Configuration configuration = null;
FileSystem hdfs = null;
FSDataOutputStream outputStream=null;
BufferedWriter br = null;
List<String> valList;
String machineValue;
int upTime;
int downTime;
int idleTime;
public void prepare(Map config, TopologyContext context,OutputCollector collector) {
upTime=0;
downTime=0;
idleTime=0;
this.collector = collector;
String timeStamp = new SimpleDateFormat("yyyyMMdd_HHmmss").format(Calendar.getInstance().getTime());
try{
configuration = new Configuration();
configuration.set("fs.defaultFS", "hdfs://localhost.localdomain:8020");
hdfs =FileSystem.get(configuration);
outputStream = hdfs.create(new Path("/tmp/storm/StormHdfs/machine10_"+timeStamp+".txt"));
br = new BufferedWriter( new OutputStreamWriter( outputStream , "UTF-8" ) );
br.flush();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public void execute(Tuple tuple) {
values = tuple.toString();
int start = values.indexOf('[');
int end = values.indexOf(']');
machineValue=values.substring(start+1,end);
String machine=machineValue.substring(0,machineValue.indexOf(','));
String code = machineValue.substring(machineValue.indexOf(',')+1);
int codeInt = Integer.parseInt(code);
if(codeInt==0) idleTime+=30;
elseif(codeInt==1) upTime+=30;
else downTime+=30;
String finalMessage = machine + " "+ "upTime(s) :" + upTime+" "+ "idleTime(s): "+idleTime+" "+"downTime: "+downTime;
try {
br.write(finalMessage); // *This is the writing part into HDFS*
br.write('\n');
br.flush();
} catch (IOException e) {
e.printStackTrace();
}
}
public void declareOutputFields(OutputFieldsDeclarer declarer) {
// this bolt does not emit anything
}
public void cleanup() {}
}
2条答案
按热度按时间nwlqm0z11#
编辑:完全改变了我的答案。
你需要使用
HdfsBolt
而不是依靠写作来归档自己。使用HdfsBolt
消除了计算何时刷新文件、打开缓冲流等的所有复杂性。请参阅http://docs.hortonworks.com/hdpdocuments/hdp2/hdp-2.1.3/bk_user-guide/content/ch_storm-using-hdfs-connector.html,但你感兴趣的是:然后只需将当前螺栓中的数据传递给此螺栓。
aamkag612#
应该使用hdfsbolt将数据插入hdfs。使用作者描述的配置。为了进行测试,应该将syncpolicy count设置为最小值(例如10-20),而不是1000。因为这个数字表示喷口发出了多少个元组,这些元组应该写在hdfs上。例如,如果您配置
然后你将能够看到数据,你已经插入Kafka后10条信息。