I am using concurrend append method from the class Core in Azure to store data to Azure Data lake.Below is the code and the exception which I got.I am getting this exception rarely not always.Could anyone guide me?...
public void invoke(String value) {
BitfinexSingletonClass obj = null;
try {
obj = BitfinexSingletonClass.getInstance();
} catch (IOException e1) {
slf4jLogger.info(e1.getMessage());
}
ADLStoreClient client = obj.getADLStoreClient();
byte[] myBuffer = (value + "\n").getBytes();
RequestOptions opts = new RequestOptions();
opts.retryPolicy = new ExponentialBackoffPolicy();
OperationResponse resp = new OperationResponse();
slf4jLogger.info("" + value);
slf4jLogger
.info("...............Writing.........above......BITFINEX_DSHBTC_ORDER..Data............................ToADLake............");
Core.concurrentAppend(BITFINEX_DSHBTC_ORDER, myBuffer, 0, myBuffer.length, true, client, opts, resp);
slf4jLogger.info("...............BITFINEX_DSHBTC_ORDER...Data...Successfully....written.....to...AzureDataLake............");
if (!resp.successful) {
try {
throw client.getExceptionFromResponse(resp, "BITFINEX_DSHBTC_ORDER data is not written to ADL");
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
com.microsoft.azure.datalake.store.adlexception:操作concurrentappend失败,出现异常java.net.sockettimeoutexception:读取超时上次尝试5次后遇到异常[java.net.unknownhostexception,java.net.unknownhostexception,java.net.unknownhostexception,java.net.sockettimeoutexception,java.net.sockettimeoutexception]位于com.microsoft.azure.datalake.store.adlstoreclient.getexceptionfromresponse(adlstoreclient)。java:1124)在co.biz.yobit.sink.yobitlctctickeradlsink.invoke(yobitlctctickeradlsink。java:41)在org.apache.flink.streaming.api.operators.streamsink.processelement(streamsink。java:38)在org.apache.flink.streaming.runtime.io.streaminputprocessor.processinput(streaminputprocessor。java:185)在org.apache.flink.streaming.runtime.tasks.oneinputstreamtask.run(oneinputstreamtask。java:63)在org.apache.flink.streaming.runtime.tasks.streamtask.invoke(streamtask。java:261)在org.apache.flink.runtime.taskmanager.task.run(task。java:665)在java.lang.thread.run(线程。java:748)原因:java.net.sockettimeoutexception:java.net.socketinputstream.socketread0(本机方法)处的java.net.socketinputstream.socketread(socketinputstream)处的读取超时。java:116)在java.net.socketinputstream.read(socketinputstream。java:171)在java.net.socketinputstream.read(socketinputstream。java:141) 在sun.security.ssl.inputrecord.readfully(inputrecord。java:465)在sun.security.ssl.inputrecord.read(inputrecord。java:503)在sun.security.ssl.sslsocketimpl.readrecord(sslsocketimpl。java:973)在sun.security.ssl.sslsocketimpl.readdatarecord(sslsocketimpl。java:930)在sun.security.ssl.appinputstream.read(appinputstream。java:105)在java.io.bufferedinputstream.fill(bufferedinputstream。java:246)在java.io.bufferedinputstream.read1(bufferedinputstream。java:286)在java.io.bufferedinputstream.read(bufferedinputstream。java:345)在sun.net。www.http.httpclient.parsehttpheader(httpclient。java:735)在sun.net。www.http.httpclient.parsehttp(httpclient。java:678)在太阳网。www.protocol.http.httpurlconnection.getinputstream0(httpurlconnection。java:1569)在sun.net。www.protocol.http.httpurlconnection.getinputstream(httpurlconnection。java:1474)在java.net.httpurlconnection.getresponsecode(httpurlconnection。java:480)在太阳网。www.protocol.https.httpsurlconnectionimpl.getresponsecode(httpsurlconnectionimpl。java:338)在com.microsoft.azure.datalake.store.httptransport.makesinglecall(httptransport)上。java:292)在com.microsoft.azure.datalake.store.httptransport.makecall(httptransport。java:91)在com.microsoft.azure.datalake.store.core.concurrentappen(core。java:210) 在co.biz.yobit.sink.yobitlctctickeradlsink.invoke(yobitlctctickeradlsink。java:37) ... 6个以上
1条答案
按热度按时间t5zmwmid1#
以上错误通常是运行代码的主机和azure数据湖存储之间不可靠的网络条件造成的。正如在评论中确认的那样,这些主机是跨地域运行的,通过广域网进行连接。因此,这些错误是意料之中的,如果您看到这些错误,应该重试
建议flink群集在与azure data lake store位于同一区域的VM上运行。在该配置中,您将不会看到这些网络错误。