incubator-doris FE 和 BE 节点正常，使用 Doris on es ，不能正确链接至 ES 地址

9cbw7uwe 于 2022-04-22 发布在 Java

关注(0)|答案(10)|浏览(533)

环境：docker 容器内容部署好单节点 BE 和 FE节点，ES 访问也是正常的，链接至扩展ES 地址,建表语句：

CREATE EXTERNAL TABLE test (
k1 bigint(20) COMMENT "",
k2 datetime COMMENT "",
k3 varchar(20) COMMENT "",
k4 varchar(100) COMMENT "",
k5 float COMMENT ""
) ENGINE=ELASTICSEARCH
PROPERTIES (
"hosts" = "http://172.17.0.1:9200",
"index" = "test",
"type" = "_doc",
"user" = "root",
"password" = "",
"enable_docvalue_scan" = "true"
);
创建语句正常执行成功，hosts 地址如果正确，但是查询的时候 Ip 地址为什么会是72.20.0.2 ，具体错误日志如下：
I0214 11:49:27.722384 15681 task_worker_pool.cpp:1098] finish report task. master host: 172.17.0.2 port: 9020
W0214 11:49:29.185159 15872 http_client.cpp:165] fail to execute HTTP client, errmsg=Failed to connect to 172.20.0.2 port 9200: Connection timed out
W0214 11:49:29.185246 15872 es_scan_reader.cpp:111] Failed to connect to ES server, errmsg is: Failed to connect to 172.20.0.2 port 9200: Connection timed out
W0214 11:49:29.185346 15872 es_http_scan_node.cpp:444] Scanner[0] process failed. status=Failed to connect to ES server, errmsg is: Failed to connect to 172.20.0.2 port 9200: Connection timed out

访问 ES 的地址变成了 172.20.0.2 这个 IP 地址。我试着把 ES 地址改为一个不可用地址会提示如下错误：
1064 - errCode = 2, detailMessage = fetch es table [test] metadata failure: Failed to connect to /172.17.0.2:9200, Time: 0.002000s

请问是否还有配置还是哪里设置不对，怎么解决 ES 的正常访问

来源：https://github.com/apache/incubator-doris/issues/5389

10条答案

按热度按时间

mrzz3bfm1#

Doris会去探测数据所在的节点，你建表的时候配置的地址是用来获取元数据信息的，通过元信息会得到数据分布的节点列表，然后直接去对应的节点上请求

赞(0）回复(0）举报 2022-04-22

flseospp2#

#5324

赞(0）回复(0）举报 2022-04-08

w7t8yxp53#

以上问题，如果我直接把 target 写成访问172.17.0.1:9200 地址，查询的时候就正常了，没有以上异常。请问下这个 target 是在配置文件中可以配置吗，我看代码目前是默认就把本机做为访问节点的。

@wuyunfeng

赞(0）回复(0）举报 2022-04-22

4jb9z9bj4#

不能手动配置哈，你的ES前面是不是有一个代理啊？Doris是不是没法访问ES的数据节点，如果是这样的话得看这个PR了：#5325

赞(0）回复(0）举报 2022-04-22

c90pui9n5#

@wuyunfeng Hi, I faced the same question, doris vesion 0.14. Table DDL：

CREATE EXTERNAL TABLE `gch_test_es` (
  `k1` bigint(20) COMMENT "",
  `k2` datetime COMMENT "",
  `k3` varchar(20) COMMENT "",
  `k4` varchar(100) COMMENT "",
  `k5` float COMMENT ""
) ENGINE=ELASTICSEARCH
PROPERTIES (
"hosts" = "http://10.50.253.1:9200,http://10.50.253.2:9200,http://10.50.253.3:9200,http://10.50.253.4:9200,http://10.50.253.5:9200,http://10.50.253.6:9200,http://10.50.253.7:9200",
"index" = "test1",
"type" = "_doc",
"user" = "",
"password" = ""
);

Some error messages appear in the be.log:

I1111 15:20:46.328352 132220 plan_fragment_executor.cpp:76] Prepare(): query_id=ab790c9841ec490b-8d831caf4787fd33 fragment_instance_id=ab790c9841ec490b-8d831caf4787fd34 backend_num=0
I1111 15:20:46.328650 132220 plan_fragment_executor.cpp:138] Using query memory limit: 2.00 GB
I1111 15:20:46.329373 131971 plan_fragment_executor.cpp:237] Open(): fragment_instance_id=ab790c9841ec490b-8d831caf4787fd34
I1111 15:20:46.329859 250195 es_scroll_query.cpp:135] Generated ES queryDSL [ {"query":{"match_all":{}},"stored_fields":"_none_","docvalue_fields":["k1","k2","k3.keyword","k4.keyword","k5"],"sort":["_doc"],"size":1024} ]
I1111 15:20:46.330041 250195 es_scan_reader.cpp:102] First scroll request URL: datanode1/10.50.253.1:9200/test1/_doc/_search?scroll=5m&preference=_shards:0&filter_path=_scroll_id,hits.total,hits.hits._score,hits.hits.fields&terminate_after=1024
W1111 15:20:46.330574 250195 http_client.cpp:165] fail to execute HTTP client, errmsg=Failed to connect to datanode1 port 80: Connection refused
W1111 15:20:46.330875 250195 es_scan_reader.cpp:111] Failed to connect to ES server, errmsg is: Failed to connect to datanode1 port 80: Connection refused
W1111 15:20:46.331095 250195 es_http_scan_node.cpp:445] Scanner[0] process failed. status=Failed to connect to ES server, errmsg is: Failed to connect to datanode1 port 80: Connection refused
W1111 15:20:46.333380 131971 fragment_mgr.cpp:220] Got error while opening fragment ab790c9841ec490b-8d831caf4787fd34: Internal error: Failed to connect to ES server, errmsg is: Failed to connect to datanode1 port 80: Connection refused
I1111 15:20:46.333977 131971 plan_fragment_executor.cpp:581] Fragment ab790c9841ec490b-8d831caf4787fd34:(Active: 1.505ms, non-child: 0.00%)
   - AverageThreadTokens: 0.00
   - FragmentCpuTime: 134.435us
   - MemoryLimit: 2.00 GB
   - PeakMemoryUsage: 8.00 KB
   - PeakReservation: 0
   - PeakUsedReservation: 0

Why es request url looks like this, so confused.

赞(0）回复(0）举报 2022-04-22

zpqajqem6#

@ddsr-ops same question， do you solve now ?

赞(0）回复(0）举报 2022-04-22

lfapxunr7#

@ddsr-ops datanode1/10.50.253.1:9200/test1/_doc/_search 确认下这个吧地址不对

赞(0）回复(0）举报 2022-04-22

b1uwtaje8#

@ddsr-ops same question， do you solve now ?

@slowmoon yes, add network.host = 0.0.0.0 in es yml config file, then restart es and recreate doris table , you can solve it .

赞(0）回复(0）举报 2022-04-22

k5ifujac9#

@ddsr-ops datanode1/10.50.253.1:9200/test1/_doc/_search 确认下这个吧地址不对

@wuyunfeng I believe you have seen the ddl of table gch_test_es, I config hosts correctly. But I add network.host = 0.0.0.0 in es yml config file, restart es, then doris could connect es successfully.

赞(0）回复(0）举报 2022-04-22

5lwkijsr10#

嗯我的意思是地址前面有个datanode1，改成0.0.0.0可以说明还是网络问题呢

赞(0）回复(0）举报 2022-04-22