incubator-doris [show frontends cost 12s]

omhiaaxx  于 2022-04-22  发布在  Java
关注(0)|答案(1)|浏览(202)

Describe the bug
Screenshots

after one FE server crash, not FE service, the show frontends become very slow

Expected behavior

show result very soon

Additional context

fe jstack log:
jstack.txt

after restart leader fe, all comes OK

ecbunoof

ecbunoof1#

i can always make it happen in my test env. 1fe , 3be . now cost40s

all costs come to "com.sleepycat.je.rep.elections.Learner$3:execute()"

`---ts=2020-11-01 03:04:21;thread_name=doris-mysql-nio-pool-53;id=12d9;is_daemon=true;priority=5;TCCL=sun.misc.Launcher$AppClassLoader@355da254
    `---[10016.334144ms] com.sleepycat.je.rep.elections.Learner:findMaster()
        +---[0.119069ms] com.sleepycat.je.rep.elections.Protocol$MasterQuery:<init>() #400
        +---[1.034996ms] com.sleepycat.je.rep.elections.Utils:broadcastMessage() #401
        +---[0.025156ms] com.sleepycat.je.rep.elections.Protocol:getReadTimeout() #408
        +---[0.036242ms] com.sleepycat.je.rep.elections.Learner$3:<init>() #408
        +---[10013.985054ms] com.sleepycat.je.rep.elections.Learner$3:execute() #432
        `---[0.027025ms] com.sleepycat.je.rep.elections.Protocol$MasterQueryResponse:getValue() #447

the slow code as bellow

static public MasterValue findMaster
        (final Protocol protocol,
         Set<InetSocketAddress> learnerSockets,
         final Logger logger,
         final RepImpl repImpl,
         final Formatter formatter )
        throws UnknownMasterException {

        if (learnerSockets.size() <= 0) {
                return null;
        }
        int threadPoolSize = Math.min(learnerSockets.size(), 10);
        final ExecutorService pool =
            Executors.newFixedThreadPool(threadPoolSize);
        try {
            FutureTrackingCompService<MessageExchange> compService =
                Utils.broadcastMessage(learnerSockets,
                                       Learner.SERVICE_NAME,
                                       protocol.new MasterQuery(),
                                       pool);

            final List<MasterQueryResponse> results = new LinkedList<>();
            new Utils.WithFutureExceptionHandler<MessageExchange>
            (compService, 2 * protocol.getReadTimeout(), TimeUnit.MILLISECONDS,
             logger, repImpl, formatter) {

                @Override
                protected void processResponse(MessageExchange me) {

                    final ResponseMessage response = me.getResponseMessage();

                    if (response.getOp() ==
                        protocol.MASTER_QUERY_RESPONSE){
                        results.add((MasterQueryResponse)response);
                    } else {
                        LoggerUtils.logMsg(logger, repImpl, formatter,
                                           Level.WARNING,
                                           "Unexpected MasterQuery response:" +
                                           response.wireFormat());
                    }
                }

                @Override
                protected boolean isShutdown() {
                    return (repImpl != null) && !repImpl.isValid();
                }

            }.execute();

            MasterQueryResponse bestResponse = null;
            for (MasterQueryResponse result : results) {
                if ((bestResponse == null) ||
                    (result.getProposal().
                        compareTo(bestResponse.getProposal()) > 0)) {
                    bestResponse = result;
                }
            }
            if (bestResponse == null) {
                throw new UnknownMasterException
                ("Could not determine master from helpers at:" +
                    learnerSockets.toString());
            }
            return(MasterValue) bestResponse.getValue();
        } finally {
            pool.shutdownNow();
        }
    }

相关问题