当我运行flink作业将数据存储到azure data lake时,我遇到以下异常有人能指导我吗?

qnakjoqk  于 2021-06-25  发布在  Flink
关注(0)|答案(1)|浏览(435)
I am using concurrend append method from the class Core in Azure to store data to Azure Data lake.Below is the code and the exception which I got.I am getting this exception rarely not always.Could anyone guide me?...

public void invoke(String value)  {
        BitfinexSingletonClass obj = null;
        try {
            obj = BitfinexSingletonClass.getInstance();
        } catch (IOException e1) {
            slf4jLogger.info(e1.getMessage());
        }
        ADLStoreClient client = obj.getADLStoreClient();
        byte[] myBuffer = (value + "\n").getBytes();

        RequestOptions opts = new RequestOptions();

        opts.retryPolicy = new ExponentialBackoffPolicy();

        OperationResponse resp = new OperationResponse();
        slf4jLogger.info("" + value);
        slf4jLogger
                .info("...............Writing.........above......BITFINEX_DSHBTC_ORDER..Data............................ToADLake............");
        Core.concurrentAppend(BITFINEX_DSHBTC_ORDER, myBuffer, 0, myBuffer.length, true, client, opts, resp);
        slf4jLogger.info("...............BITFINEX_DSHBTC_ORDER...Data...Successfully....written.....to...AzureDataLake............");
        if (!resp.successful) {
            try {
                throw client.getExceptionFromResponse(resp, "BITFINEX_DSHBTC_ORDER data is not written to ADL");
            } catch (IOException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        }

    }

com.microsoft.azure.datalake.store.adlexception:操作concurrentappend失败,出现异常java.net.sockettimeoutexception:读取超时上次尝试5次后遇到异常[java.net.unknownhostexception,java.net.unknownhostexception,java.net.unknownhostexception,java.net.sockettimeoutexception,java.net.sockettimeoutexception]位于com.microsoft.azure.datalake.store.adlstoreclient.getexceptionfromresponse(adlstoreclient)。java:1124)在co.biz.yobit.sink.yobitlctctickeradlsink.invoke(yobitlctctickeradlsink。java:41)在org.apache.flink.streaming.api.operators.streamsink.processelement(streamsink。java:38)在org.apache.flink.streaming.runtime.io.streaminputprocessor.processinput(streaminputprocessor。java:185)在org.apache.flink.streaming.runtime.tasks.oneinputstreamtask.run(oneinputstreamtask。java:63)在org.apache.flink.streaming.runtime.tasks.streamtask.invoke(streamtask。java:261)在org.apache.flink.runtime.taskmanager.task.run(task。java:665)在java.lang.thread.run(线程。java:748)原因:java.net.sockettimeoutexception:java.net.socketinputstream.socketread0(本机方法)处的java.net.socketinputstream.socketread(socketinputstream)处的读取超时。java:116)在java.net.socketinputstream.read(socketinputstream。java:171)在java.net.socketinputstream.read(socketinputstream。java:141) 在sun.security.ssl.inputrecord.readfully(inputrecord。java:465)在sun.security.ssl.inputrecord.read(inputrecord。java:503)在sun.security.ssl.sslsocketimpl.readrecord(sslsocketimpl。java:973)在sun.security.ssl.sslsocketimpl.readdatarecord(sslsocketimpl。java:930)在sun.security.ssl.appinputstream.read(appinputstream。java:105)在java.io.bufferedinputstream.fill(bufferedinputstream。java:246)在java.io.bufferedinputstream.read1(bufferedinputstream。java:286)在java.io.bufferedinputstream.read(bufferedinputstream。java:345)在sun.net。www.http.httpclient.parsehttpheader(httpclient。java:735)在sun.net。www.http.httpclient.parsehttp(httpclient。java:678)在太阳网。www.protocol.http.httpurlconnection.getinputstream0(httpurlconnection。java:1569)在sun.net。www.protocol.http.httpurlconnection.getinputstream(httpurlconnection。java:1474)在java.net.httpurlconnection.getresponsecode(httpurlconnection。java:480)在太阳网。www.protocol.https.httpsurlconnectionimpl.getresponsecode(httpsurlconnectionimpl。java:338)在com.microsoft.azure.datalake.store.httptransport.makesinglecall(httptransport)上。java:292)在com.microsoft.azure.datalake.store.httptransport.makecall(httptransport。java:91)在com.microsoft.azure.datalake.store.core.concurrentappen(core。java:210) 在co.biz.yobit.sink.yobitlctctickeradlsink.invoke(yobitlctctickeradlsink。java:37) ... 6个以上

t5zmwmid

t5zmwmid1#

以上错误通常是运行代码的主机和azure数据湖存储之间不可靠的网络条件造成的。正如在评论中确认的那样,这些主机是跨地域运行的,通过广域网进行连接。因此,这些错误是意料之中的,如果您看到这些错误,应该重试
建议flink群集在与azure data lake store位于同一区域的VM上运行。在该配置中,您将不会看到这些网络错误。

相关问题