如何将python螺栓集成到使用storm crawler sdk构建的拓扑中

qacovj5a  于 2021-06-24  发布在  Storm
关注(0)|答案(1)|浏览(382)

我试图在使用storm-crawler\usdk-1.7和apache-storm-1.1.0组件构建的拓扑中集成一个用python创建的bolt。拓扑执行无法找到可执行的python程序并在完全不同的临时位置搜索它。我在尝试执行拓扑时经常会遇到以下错误:

27238 [Thread-20-classify-executor[2 2]] ERROR o.a.s.util - Async loop died!
java.lang.RuntimeException: Error when launching multilang subprocess

    at org.apache.storm.utils.ShellProcess.launch(ShellProcess.java:94) ~[storm-core-1.1.1.jar:1.1.1]
    at org.apache.storm.task.ShellBolt.prepare(ShellBolt.java:150) ~[storm-core-1.1.1.jar:1.1.1]
    at org.apache.storm.daemon.executor$fn__5030$fn__5043.invoke(executor.clj:793) ~[storm-core-1.1.1.jar:1.1.1]
    at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:482) [storm-core-1.1.1.jar:1.1.1]
    at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
    at java.lang.Thread.run(Unknown Source) [?:1.8.0_171]
Caused by: java.io.IOException: Cannot run program "python" (in directory "C:\Users\akumar\AppData\Local\Temp\24ed7755-e7c0-42d4-a17f-939082feb1a8\supervisor\stormdist\crawler-1-1528614625\resources"): CreateProcess error=267, The directory name is inval*id
    at java.lang.ProcessBuilder.start(Unknown Source) ~[?:1.8.0_171]
    at org.apache.storm.utils.ShellProcess.launch(ShellProcess.java:87) ~[storm-core-1.1.1.jar:1.1.1]
    ... 5 more
Caused by: java.io.IOException: CreateProcess error=267, The directory name is invalid
    at java.lang.ProcessImpl.create(Native Method) ~[?:1.8.0_171]
    at java.lang.ProcessImpl.<init>(Unknown Source) ~[?:1.8.0_171]
    at java.lang.ProcessImpl.start(Unknown Source) ~[?:1.8.0_171]
    at java.lang.ProcessBuilder.start(Unknown Source) ~[?:1.8.0_171]
    at org.apache.storm.utils.ShellProcess.launch(ShellProcess.java:87) ~[storm-core-1.1.1.jar:1.1.1]
    ... 5 more
27246 [Thread-20-classify-executor[2 2]] ERROR o.a.s.d.executor - 
java.lang.RuntimeException: Error when launching multilang subprocess

    at org.apache.storm.utils.ShellProcess.launch(ShellProcess.java:94) ~[storm-core-1.1.1.jar:1.1.1]
    at org.apache.storm.task.ShellBolt.prepare(ShellBolt.java:150) ~[storm-core-1.1.1.jar:1.1.1]
    at org.apache.storm.daemon.executor$fn__5030$fn__5043.invoke(executor.clj:793) ~[storm-core-1.1.1.jar:1.1.1]
    at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:482) [storm-core-1.1.1.jar:1.1.1]
    at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
    at java.lang.Thread.run(Unknown Source) [?:1.8.0_171]
Caused by: java.io.IOException: Cannot run program "python" (in directory "C:\Users\akumar\AppData\Local\Temp\24ed7755-e7c0-42d4-a17f-939082feb1a8\supervisor\stormdist\crawler-1-1528614625\resources"): CreateProcess error=267, The directory name is invalid
    at java.lang.ProcessBuilder.start(Unknown Source) ~[?:1.8.0_171]
    at org.apache.storm.utils.ShellProcess.launch(ShellProcess.java:87) ~[storm-core-1.1.1.jar:1.1.1]
    ... 5 more
Caused by: java.io.IOException: CreateProcess error=267, The directory name is invalid
    at java.lang.ProcessImpl.create(Native Method) ~[?:1.8.0_171]
    at java.lang.ProcessImpl.<init>(Unknown Source) ~[?:1.8.0_171]
    at java.lang.ProcessImpl.start(Unknown Source) ~[?:1.8.0_171]
    at java.lang.ProcessBuilder.start(Unknown Source) ~[?:1.8.0_171]
    at org.apache.storm.utils.ShellProcess.launch(ShellProcess.java:87) ~[storm-core-1.1.1.jar:1.1.1]
    ... 5 more
27624 [Thread-30-sitemap-executor[9 9]] INFO  c.d.s.f.URLFilters - Loaded instance of class com.digitalpebble.stormcrawler.filtering.regex.RegexURLFilter
27626 [Thread-30-sitemap-executor[9 9]] INFO  o.a.s.d.executor - Prepared bolt sitemap:(9)
27627 [Thread-40-fetch-executor[4 4]] INFO  c.d.s.f.URLFilters - Loaded instance of class com.digitalpebble.stormcrawler.filtering.regex.RegexURLFilter
27628 [Thread-40-fetch-executor[4 4]] INFO  c.d.s.b.FetcherBolt - [Fetcher #-1] : starting at 2018-06-10 12:40:37
27638 [Thread-24-feed-executor[3 3]] INFO  c.d.s.f.URLFilters - Loaded instance of class com.digitalpebble.stormcrawler.filtering.regex.RegexURLFilter
27639 [Thread-24-feed-executor[3 3]] INFO  o.a.s.d.executor - Prepared bolt feed:(3)
27661 [Thread-20-classify-executor[2 2]] ERROR o.a.s.util - Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
    at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [storm-core-1.1.1.jar:1.1.1]
    at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
    at org.apache.storm.daemon.worker$fn__5628$fn__5629.invoke(worker.clj:759) [storm-core-1.1.1.jar:1.1.1]
    at org.apache.storm.daemon.executor$mk_executor_data$fn__4848$fn__4849.invoke(executor.clj:276) [storm-core-1.1.1.jar:1.1.1]
    at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:494) [storm-core-1.1.1.jar:1.1.1]
    at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
    at java.lang.Thread.run(Unknown Source) [?:1.8.0_171]

当python螺栓不包含在拓扑中时,拓扑工作正常。
当我没有在拓扑中使用storm crawler sdk组件时,PythonBolt也能像预期的那样工作。
有人能帮忙吗?

l3zydbqr

l3zydbqr1#

当我为ApacheStorm核心依赖选项更改maven的pom.xml配置时,问题就解决了。
标签设置为“提供”。我将它改为compile,解决了在本地模式下运行的问题。

相关问题