tengine dyups upstream unstable

nukf8bse  于 4个月前  发布在  其他
关注(0)|答案(7)|浏览(128)

Hi

When we do:
curl http://localhost:8081/detail

we see the upstream list changes from call to call ?
when creating upstreams

in tengine log I can see:
2019/01/09 19:37:07 [info] 42#0: [dyups] has 3 upstreams, 0 static, 0 deleted, all 3
2019/01/09 19:37:07 [info] 71#0: [dyups] has 1 upstreams, 0 static, 1 deleted, all 2
2019/01/09 19:37:07 [info] 41#0: [dyups] has 3 upstreams, 0 static, 0 deleted, all 3

jfewjypa

jfewjypa1#

tengine version is 2.2.2

8aqjt8rx

8aqjt8rx2#

also see some memory error in tengine log:
2019/01/09 20:04:02 [info] 42#0: *259 [dyups] post upstream name: elasticsearch, client: 127.0.0.1, server: , request:
"POST /upstream/elasticsearch HTTP/1.1", host: "localhost:8081"
2019/01/09 20:04:02 [info] 42#0: [dyups] create upstream dyups_upstream_sandbox
2019/01/09 20:04:02 [crit] 42#0: ngx_slab_alloc() failed: no memory
2019/01/09 20:04:02 [notice] 39#0: signal 17 (SIGCHLD) received
2019/01/09 20:04:02 [alert] 39#0: worker process 42 exited on signal 11 (core dumped)
2019/01/09 20:04:02 [alert] 39#0: shared memory zone "ngx_http_upstream_check#1" was locked by 42
2019/01/09 20:04:02 [alert] 39#0: shared memory zone "ngx_http_dyups_module#1" was locked by 42

kknvjkwl

kknvjkwl3#

Dyups works ok in 2.2.0, but not in 2.2.1 and 2.2.2 for me

lp0sw83n

lp0sw83n4#

Seem to be related to no of servers when calling http://localhost:8081/upstream
If I call with 6 servers it works ok, but when using 7 it fails:

server x.x.x.1:8080;server x.x.x.2:8080;server x.x.x.3:8080;server x.x.x.4:8080;server x.x.x.5:8080;server x.x.x.6:8080
this works ok

server x.x.x.1:8080;server x.x.x.2:8080;server x.x.x.3:8080;server x.x.x.4:8080;server x.x.x.5:8080;server x.x.x.6:8080; server x.x.x.7:8080

fails with this after posting to /upstream:
'Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

k4aesqcs

k4aesqcs5#

Error happens after about 4 calls to /upstream and then starts to what seems core dump in nginx, when calling:
curl -v --data @test2.txt localhost:8081/upstream/test

  • Trying 127.0.0.1...

  • Connected to localhost (127.0.0.1) port 8081 (#0)
    POST /upstream/test HTTP/1.1
    Host: localhost:8081
    User-Agent: curl/7.47.0
    Accept: /
    Content-Length: 366
    Content-Type: application/x-www-form-urlencoded

  • upload completely sent off: 366 out of 366 bytes

  • Empty reply from server

  • Connection #0 to host localhost left intact
    curl: (52) Empty reply from server

same call have worked initial before, so seems like a memory issue as per below debug lines in nginx error-log for above curl call:

019/01/24 21:04:23 [debug] 95#0: *1112 [dyups] interface read post body
2019/01/24 21:04:23 [info] 95#0: *1112 [dyups] post upstream name: test, client: 127.0.0.1, server: , request: "POST /upstream/test HTTP/1.1", host: "localhost:8081"
2019/01/24 21:04:23 [debug] 95#0: shmtx lock
2019/01/24 21:04:23 [debug] 95#0: [dyups] read msg 95
2019/01/24 21:04:23 [debug] 95#0: [dyups] process 95 update time 1548363863209
2019/01/24 21:04:23 [debug] 95#0: [dyups] find dynamic upstream
2019/01/24 21:04:23 [info] 95#0: [dyups] create upstream dyups_upstream_sandbox
2019/01/24 21:04:23 [debug] 95#0: posix_memalign: 00000000016D5080:512 @16
2019/01/24 21:04:23 [debug] 95#0: malloc: 00000000016DD9E0:512
2019/01/24 21:04:23 [debug] 95#0: posix_memalign: 000000000162D4E0:512 @16
2019/01/24 21:04:23 [debug] 95#0: posix_memalign: 000000000162D6F0:512 @16
2019/01/24 21:04:23 [debug] 95#0: posix_memalign: 000000000162D900:512 @16
2019/01/24 21:04:23 [debug] 95#0: posix_memalign: 000000000162DB10:512 @16
2019/01/24 21:04:23 [debug] 95#0: malloc: 00000000016E0C50:768
2019/01/24 21:04:23 [debug] 95#0: posix_memalign: 00000000016E0F60:512 @16
2019/01/24 21:04:23 [debug] 95#0: malloc: 00000000016E1170:1536
2019/01/24 21:04:23 [debug] 95#0: posix_memalign: 00000000016E1FF0:512 @16
2019/01/24 21:04:23 [debug] 95#0: init least conn
2019/01/24 21:04:23 [debug] 95#0: malloc: 00000000016E2200:1560
2019/01/24 21:04:23 [debug] 95#0: http upstream check add upstream process: 3
2019/01/24 21:04:23 [debug] 95#0: shmtx lock
2019/01/24 21:04:24 [notice] 84#0: signal 17 (SIGCHLD) received
2019/01/24 21:04:24 [alert] 84#0: worker process 95 exited on signal 11 (core dumped)

xriantvc

xriantvc6#

Have identified the issue starts in commitid 2ece5b2 and works ok in previous 82ebf09

kadbb459

kadbb4597#

You could try the latest version of Tengine-2.3.0 or Tengine-2.3.1.

相关问题