我正在为cloudera的分布式日志聚合系统flume编写一个定制的decorator插件。我的java代码如下:
package multiplex;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import com.cloudera.flume.conf.Context;
import com.cloudera.flume.conf.SinkFactory.SinkDecoBuilder;
import com.cloudera.flume.core.Event;
import com.cloudera.flume.core.EventImpl;
import com.cloudera.flume.core.EventSink;
import com.cloudera.flume.core.EventSinkDecorator;
import com.cloudera.util.Pair;
import com.google.common.base.Preconditions;
public class JsonMultiplexDecorator<S extends EventSink> extends EventSinkDecorator<S> {
private final String serverName;
private final String logType;
public JsonMultiplexDecorator(S s, String serverName, String logType) {
super(s);
this.serverName = serverName;
this.logType = logType;
}
@Override
public void append(Event e) throws IOException {
String body = new String(e.getBody()).replaceAll("\"", "\\\"");
String json = "{ \"server\": \"" + this.serverName + "\"," +
"\"log_type\": \"" + this.logType + "\", " +
"\"body\": \"" + body + "\" }";
EventImpl e2 = new EventImpl(json.getBytes(),
e.getTimestamp(), e.getPriority(), e.getNanos(), e.getHost(),
e.getAttrs());
super.append(e2);
}
public static SinkDecoBuilder builder() {
return new SinkDecoBuilder() {
@Override
public EventSinkDecorator<EventSink> build(Context context,
String... argv) {
Preconditions.checkArgument(argv.length == 2,
"usage: multiplexDecorator(serverName, logType)");
return new JsonMultiplexDecorator<EventSink>(null, argv[0], argv[1]);
}
};
}
public static List<Pair<String, SinkDecoBuilder>> getDecoratorBuilders() {
List<Pair<String, SinkDecoBuilder>> builders =
new ArrayList<Pair<String, SinkDecoBuilder>>();
builders.add(new Pair<String, SinkDecoBuilder>("jsonMultiplexDecorator", builder()));
return builders;
}
}
这可以很好地用ant编译成jar文件,我可以在运行时将它加载到flume中,并成功地配置节点来使用它。但是,当一个事件实际发生在加载了这个插件的节点上时,我的日志中会出现如下错误:
2010-10-19 21:03:15,176 [logicalNode xxxxx] ERROR connector.DirectDriver: Driving src/sink failed! LazyOpenSource | LazyOpenDecorator because null
java.lang.UnsupportedOperationException
at java.util.Collections$UnmodifiableMap.put(Collections.java:1285)
at com.cloudera.flume.core.EventBaseImpl.set(EventBaseImpl.java:65)
at com.cloudera.flume.handlers.rolling.RollSink.append(RollSink.java:164)
at com.cloudera.flume.agent.diskfailover.DiskFailoverDeco.append(DiskFailoverDeco.java:93)
at com.cloudera.flume.core.BackOffFailOverSink.append(BackOffFailOverSink.java:144)
at com.cloudera.flume.agent.AgentSink.append(AgentSink.java:109)
at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:58)
at multiplex.JsonMultiplexDecorator.append(JsonMultiplexDecorator.java:56)
at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:58)
at com.cloudera.flume.handlers.debug.LazyOpenDecorator.append(LazyOpenDecorator.java:69)
at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:92)
(the) [logicalNode xx]
是ec2内部dns名称的占位符)。我没有太多的java经验,所以我不确定我在这里是否做错了什么,或者这是一个flume bug。我应该提到,我写这篇文章时使用了flume源代码中的helloworld插件示例,还使用了一些内置flume装饰器。
1条答案
按热度按时间vnzz0bqm1#
当你建造
EventImpl e2
,你路过e.getAttrs()
,不可修改。尝试复制e.getAttrs()
变成你自己的Map;使用new HashMap(e.getAttrs())
应该足够了。参考文献:https://groups.google.com/a/cloudera.org/group/flume-user/browse_thread/thread/046b4a446877c8f9?pli=1