连接到java中aws emr集群上的hbase

我正在尝试访问aws emr群集上的hbase，但唯一得到的是：

2018-04-24 18:53:29 WARN  org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Operation timed out
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

我正在尝试通过以下方式访问它：

import com.google.protobuf.ServiceException;
import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.HBaseAdmin;

import java.io.IOException;

public class HBaseEMR {

    public static void main(String[] args) throws ServiceException, IOException {
        Configuration conf = HBaseConfiguration.create();
        conf.set("hbase.zookeeper.quorum","ec2-123456789101112.us-west-2.compute.amazonaws.com");
        conf.set("hbase.zookeeper.property.clientPort","2181");
        conf.set("hbase.master","ec2-123456789101112.us-west-2.compute.amazonaws.com");
        conf.set("hbase.master.port","60000");
        conf.set("hbase.rootdir", "s3://bucket/");

        HBaseAdmin.checkHBaseAvailable(conf);
    }
}

我修改了安全组，使通过端口22的ssh连接可用，但仍然无法工作。欢迎各位指教！
编辑：将端口2181添加到elasticmapreduce主机的安全组规则后，出现以下错误：

2018-04-25 15:51:28 INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server ec2-<myDNS>.us-west-2.compute.amazonaws.com/<myDNS>:2181. Will not attempt to authenticate using SASL (unknown error)
2018-04-25 15:51:28 INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to ec2-<myDNS>.us-west-2.compute.amazonaws.com/<myDNS>:2181, initiating session
2018-04-25 15:51:28 INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server ec2-<myDNS>.us-west-2.compute.amazonaws.com/<myDNS>:2181, sessionid = 0x162f882
2018-04-25 15:51:30 INFO  org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation - Closing zookeeper sessionid=0x162f88235d80010
2018-04-25 15:51:30 INFO  org.apache.zookeeper.ZooKeeper - Session: 0x162f88235d80010 closed
2018-04-25 15:51:30 INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down
Exception in thread "main" org.apache.hadoop.hbase.MasterNotRunningException: java.net.UnknownHostException: ip-172-31-11-153.us-west-2.compute.internal
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1561)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1581)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1738)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.isMasterRunning(ConnectionManager.java:948)
    at org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:3162)
    at twitter.HBaseEMR.main(HBaseEMR.java:33)

请看一下这个代码。我用这个连接到hbase。我想你错过了“zookeeper.znode.parent”属性。

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.log4j.Logger;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import com.naresh.common.util.PropertyUtil;

/**
 * 
 * @author Naresh Bharadwaj
 * This class contains functionality for Creating Hbase Connection
 * 
 */
@Component
public class HbaseConnection {

    private static final Logger _LOGGER = Logger.getLogger(HbaseConnection.class);

    @Autowired
    private PropertyUtil propertyUtil;

    private Boolean isKerberosEnabled;

    public Connection getConnection() throws IOException {

        Configuration conf = HBaseConfiguration.create();

                //Base 
        conf.set("hbase.zookeeper.quorum", propertyUtil.getValue("hbase.zookeeper.quorum"));
        conf.set("hbase.zookeeper.property.clientPort", propertyUtil.getValue("hbase.zookeeper.property.clientPort"));
        conf.set("zookeeper.znode.parent", propertyUtil.getValue("hbase.zookeeper.znode.parent"));

        return ConnectionFactory.createConnection(conf);
    }

    public void closeConnection(Connection connection) {
        try {
            connection.close();
        } catch (IOException e) {
            _LOGGER.error("Error while closing connection ", e);
        }
    }

}

连接到java中aws emr集群上的hbase

1条答案

相关问题

热门标签

最新问答