通过java连接到一个遥远的hadoop集群(cdh4)

w6lpcovy  于 2021-06-03  发布在  Hadoop
关注(0)|答案(0)|浏览(268)

我有一个遥远的hadoop机器集群(clouderacdh4)。我正试着从我的电脑上运行一个Pig脚本。下面是我的java代码:

import org.apache.pig.ExecType;
import org.apache.pig.PigServer;
import org.apache.pig.backend.executionengine.ExecException;
import org.apache.pig.data.Tuple;

public class TestPig {

public static void main(String args[]){

    PigServer pigServer;
    try {

        /**On définit les propriétés */
        Properties props = new Properties();

        props.setProperty("fs.default.name", "hdfs://master.node.ip.adress:8020");
        props.setProperty("mapred.job.tracker", "master.node.ip.adress:8021");

        System.setProperty("javax.xml.parsers.DocumentBuilderFactory", "com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl");

        /**mapreduce mode */
        pigServer = new PigServer(ExecType.MAPREDUCE, props);

        /**pig script's path */
        pigServer.registerScript("/user/admin/data/script.pig");

        /**printing the pig script's output */
        Iterator<Tuple> results = pigServer.openIterator("A");
        while(results.hasNext())
            System.out.println(results.next().toDelimitedString("\t"));

    } 
    catch (ExecException e) {   e.printStackTrace(); } 
    catch (IOException e) { e.printStackTrace(); }

}
}

启动此程序时,出现以下错误:
19年7月13日10:44:02 info executionengine.hexecutionengine:连接到hadoop文件系统的位置:hdfs://master.node.ip.adress:8020 13/07/19 10:44:23信息ipc.client:正在重试连接到服务器:master.cs236cloud.internal/master.node.ip。adress:8020. 已尝试0次;最大重试次数=45
...
19年7月13日10:59:27信息ipc.client:正在重试连接到服务器:master.cs236cloud.internal/master.node.ip。adress:8020. 已尝试43次;maxretries=45 13/07/19 10:59:48 info ipc.client:正在重试连接到服务器:master.cs236cloud.internal/master.node.ip。adress:8020. 已尝试44次;最大重试次数=45
线程“main”java.lang.runtimeexception中出现异常:未能在org.apache.pig.backend.hadoop.datastorage.hdatastorage.init(hdatastorage)中创建数据存储。java:75)在org.apache.pig.backend.hadoop.datastorage.hdatastorage.(hdatastorage。java:58)位于org.apache.pig.backend.hadoop.executionengine.hexecutionengine.init(hexecutionengine。java:204) 位于org.apache.pig.backend.hadoop.executionengine.hexecutionengine.init(hexecutionengine。java:117)在org.apache.pig.impl.pigcontext.connect(pigcontext。java:240)在org.apache.pig.pigserver.(pigserver。java:213)在org.apache.pig.pigserver.(pigserver。java:198)在org.apache.pig.pigserver.(pigserver。java:194)在testpig.main(testpig。java:42)引起作者:java.net.sockettimeoutexception:调用master.cs236cloud.internal/master.node.ip。adress:8020 failed 套接字超时异常:java.net.sockettimeoutexception:等待通道准备好连接时超时20000毫秒。ch:java.nio.channels.socketchannel[connection pending remote=master.cs236cloud.internal/master.node.ip。adress:8020]在org.apache.hadoop.ipc.client.wrapexception(client。java:1140)在org.apache.hadoop.ipc.client.call(client。java:1112)在org.apache.hadoop.ipc.rpc$invoker.invoke(rpc。java:229)在$proxy1.getprotocolversion(未知源)处sun.reflect.nativemethodaccessorimpl.invoke0(本机方法)位于sun.reflect.nativemethodaccessorimpl.invoke(未知源)位于java.lang.reflect.method.invoke(未知源)位于org.apache.hadoop.io.retry.retryinvocationhandler.invokemethod(retryinvocationhandler)。java:85)在org.apache.hadoop.io.retry.retryinvocationhandler.invoke(retryinvocationhandler。java:62)$proxy1.getprotocolversion(未知源代码),org.apache.hadoop.ipc.rpc.checkversion(rpc。java:422)在org.apache.hadoop.hdfs.dfsclient.createnamenode(dfsclient。java:183)在org.apache.hadoop.hdfs.dfsclient。java:281)在org.apache.hadoop.hdfs.dfsclient.(dfsclient。java:245)位于org.apache.hadoop.hdfs.distributedfilesystem.initialize(distributedfilesystem)。java:100)在org.apache.hadoop.fs.filesystem.createfilesystem(filesystem。java:1437)在org.apache.hadoop.fs.filesystem.access$200(文件系统)。java:66)在org.apache.hadoop.fs.filesystem$cache.get(文件系统)。java:1455)在org.apache.hadoop.fs.filesystem.get(filesystem。java:254)在org.apache.hadoop.fs.filesystem.get(filesystem。java:123)位于org.apache.pig.backend.hadoop.datastorage.hdatastorage.init(hdatastorage)。java:72) ... 另外8个原因:java.net.sockettimeoutexception:等待通道准备好连接时超时20000毫秒。ch:java.nio.channels.socketchannel[connection pending remote=master.cs236cloud.internal/master.node.ip。adress:8020]在org.apache.hadoop.net.socketiowithtimeout.connect(socketiowithtimeout)上。java:213)在org.apache.hadoop.net.netutils.connect(netutils。java:511)在org.apache.hadoop.net.netutils.connect(netutils。java:481)在org.apache.hadoop.ipc.client$connection.setupconnection(客户端。java:453)在org.apache.hadoop.ipc.client$connection.setupiostreams(client。java:579)在org.apache.hadoop.ipc.client$connection.access$2100(客户端。java:202)在org.apache.hadoop.ipc.client.getconnection(client。java:1243)在org.apache.hadoop.ipc.client.call(client。java:1087) ... 28个以上
我肯定船长在8020端口监听。实际上,当我启动这个命令时:
netstat-一个| grep 8020
我得到了这个结果:
tcp 0 0 master.local.ip。adress:8020 0.0.0.0:*听着
我还想给你我的core-site.xml文件:

<?xml version="1.0" encoding="UTF-8"?>
<!--Autogenerated by Cloudera CM on 2013-07-12T10:43:15.666Z-->
<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://Master.cs236cloud.internal:8020</value>
  </property>
  <property>
    <name>fs.trash.interval</name>
    <value>1</value>
  </property>
  <property>
    <name>io.file.buffer.size</name>
    <value>65536</value>
  </property>
  <property>
    <name>io.compression.codecs</name>
       <value>org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.DeflateCodec,org.apache.hadoop.io.compress.SnappyCodec,org.apache.hadoop.io.compress.Lz4Codec</value>
  </property>
  <property>
    <name>hadoop.security.authentication</name>
    <value>simple</value>
  </property>
  <property>
    <name>hadoop.rpc.protection</name>
    <value>authentication</value>
  </property>
  <property>
    <name>hadoop.security.auth_to_local</name>
    <value>DEFAULT</value>
  </property>
</configuration>

以下是在我的主计算机上执行jps命令的结果:

3250 DataNode
4468 Main
2776 HeadlampServer
2541 RunJar
5496 Jps
4467 EventCatcherService
2502 QuorumPeerMain
2650 JobTracker
3082 RunJar
2597 HRegionServer
2594 TaskTracker
2629 HMaster
2520
3003 SecondaryNameNode
4553 Main
3414 Bootstrap
2549 AlertPublisher
3172 NameNode
2127 Main
4583 Main
3350 Bootstrap

我在网上搜索解决方案,但没有找到任何有效的方法。
你有办法解决这个问题吗?
谢谢。

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题