无法提交storm拓扑

odopli94 于 2021-06-21 发布在 Storm

关注(0)|答案(2)|浏览(371)

我正在尝试使用eclipse提交远程主机上的storm拓扑。
这是我的密码：

Config conf = new Config();
conf.setDebug(false);
conf.setNumWorkers(1);
conf.put(Config.NIMBUS_HOST, "hostName");
conf.put(Config.NIMBUS_THRIFT_PORT,6627);
conf.put(Config.STORM_ZOOKEEPER_SERVERS,Arrays.asList(new String[]{"hostName"}));
conf.put(Config.STORM_ZOOKEEPER_PORT,2181);

// Remote submission
StormSubmitter.submitTopology("classMain", conf, topology);

但我有个例外：

Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.TApplicationException: Binary field exceeded string size limit
  at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:250)
 at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)
  at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)
  at com.rbc.rbccm.hackathon.Countersearch.submitTopology(Countersearch.java:111)
  at com.rbc.rbccm.hackathon.Countersearch.main(Countersearch.java:37)
Caused by: org.apache.thrift7.TApplicationException: Binary field exceeded string size limit
  at org.apache.thrift7.TApplicationException.read(TApplicationException.java:111)
  at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:71)
  at backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:184)
  at backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:168)
  at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:236)
... 4 more

我们可以传递给submittopology函数的参数是否有字符串大小限制？
当我沿着这条小径走多一点的时候，它会导致：

public void submitTopology(String name, String uploadedJarLocation, String jsonConf, StormTopology topology) throws AlreadyAliveException, InvalidTopologyException, AuthorizationException, org.apache.thrift.TException
{
    send_submitTopology(name, uploadedJarLocation, jsonConf, topology);
    recv_submitTopology();
}

这个 recv 是问题的根源。有什么想法吗？

Java apache-storm

来源：https://stackoverflow.com/questions/33172107/unable-to-submit-storm-topology

2条答案

按热度按时间

jv4diomz1#

你需要增加 nimbus.thrift.max_buffer_size 参数。你可以把它放进去 storm.yaml 或在 Config 对象。

赞(0）回复(0）举报 2021-06-21

mzillmmw2#

如果你在storm的源代码中看到代码 StormSubmitter.java ，是这样的：

public static void submitTopology(String name, Map stormConf, StormTopology topology)
        throws AlreadyAliveException, InvalidTopologyException, AuthorizationException {
    submitTopology(name, stormConf, topology, null, null);
}

节俭的错误是因为 name 您指定的太长（超过2mb？）或 stormConf 有太多的信息或者更有可能的原因，那就是 topology 创建时，您正在用太多的信息填充喷口或螺栓示例。
在我的例子中，我创建了一个螺栓，我初始化了太多的数据。

builder.setBolt(genBolt, new GenBolt(myTable1.getHashMap(), myTable2.getHashMap(), myTable3.getHashMap(), myTable4.getHashMap()), 2)
  .fieldsGrouping(iterSpout, new Fields(con.BATCH_ID))

赞(0）回复(0）举报 2021-06-21

我来回答

无法提交storm拓扑

2条答案

相关问题

热门标签

最新问答