我的目标:
使用运行在hdf docker容器上的nifi将数据存储到运行在hdp docker容器上的hbase中。
进展:
我运行两个docker容器:nifi和hbase。我已经配置了nifi的puthbasejson处理器将数据写入hbase(puthbasejson_configuration.png)。以下是我在处理器中更新的配置:
PutHBaseJSON = HBase_1_1_2_ClientService
Table Name = publictrans
Row Identifier Field Name = Vehicle_ID
Row Identifier Encoding Strategy = String
Column Family = trafficpatterns
Batch Size = 25
Complex Field Strategy = Text
Field Encoding Strategy = String
hbase客户端服务
我还在该处理器上配置了nifi的hbase客户机服务,以便nifi知道zookeeper位于哪个ip地址,让zookeeper告诉它hbase master在哪里(hbaseclientservice\ u configuration.png)。
为hbaseclient配置控制器服务:
ZooKeeper Quorum = 172.25.0.3
ZooKeeper Client Port = 2181
ZooKeeper ZNode Parent = /hbase-unsecure
HBase Client Retries = 1
问题:
我面临的问题是nifi无法连接到hbase master。我收到以下消息:调用“@onenabled方法失败,原因是。。。hbase.client.retriesExhaused异常。。。hbase.masternotrunningexception。。。java.net.connectexception:连接被拒绝。“hbaseclientservice的可视位置(hbasemasternotrunningexception堆栈跟踪)。
我为解决问题所做的配置:
在hdf容器中,我用172.25.0.3->hdp.hortonworks.com更新了/etc/hosts。在hdp容器中,我用172.25.0.2->hdf.hortonworks.com更新了hosts文件。所以两个容器都知道彼此的主机名。
在构建hdf和hdp容器时,我为nifi、zookeeper和hbase转发了所需的端口。我检查了hbase上的所有端口是否都暴露在hdp容器中,并且图像显示了hdp正在侦听的所有端口,包括hbase的端口(ports\u hdp\u listening\u on.png)。这是hbase需要的所有端口的图像,我在ambari中过滤了port关键字(hbase\u ports\u needed.png)。
16000和16020个港口看起来都可疑,因为所有其他港口都有模式:::港口,但这两个港口之前有一些措辞。因此,我检查了是否可以使用telnet 172.25.0.3 16000从hdf连接到hdp,并收到了输出:
正在尝试172.25.0.3。。。
接172.25.0.3。
转义字符为“^]”。
所以我可以连接到hdp容器。
hbasemasternotrunningexception堆栈跟踪:
2017-01-25 22:23:03,342 ERROR [StandardProcessScheduler Thread-7] o.a.n.c.s.StandardControllerServiceNode HBase_1_1_2_ClientService[id=d3eaf393-0159-1000-ffff-ffffa95f1940] Failed to invoke @OnEnabled method due to org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions:
Wed Jan 25 22:23:03 UTC 2017, RpcRetryingCaller{globalStartTime=1485382983338, pause=100, retries=1}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.net.ConnectException: Connection refused
2017-01-25 22:23:03,348 ERROR [StandardProcessScheduler Thread-7] o.a.n.c.s.StandardControllerServiceNode
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions:
Wed Jan 25 22:23:03 UTC 2017, RpcRetryingCaller{globalStartTime=1485382983338, pause=100, retries=1}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.net.ConnectException: Connection refused
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147) ~[na:na]
at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3917) ~[na:na]
at org.apache.hadoop.hbase.client.HBaseAdmin.listTableNames(HBaseAdmin.java:413) ~[na:na]
at org.apache.hadoop.hbase.client.HBaseAdmin.listTableNames(HBaseAdmin.java:397) ~[na:na]
at org.apache.nifi.hbase.HBase_1_1_2_ClientService.onEnabled(HBase_1_1_2_ClientService.java:187) ~[na:na]
at sun.reflect.GeneratedMethodAccessor568.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111]
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137) ~[na:na]
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125) ~[na:na]
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70) ~[na:na]
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47) ~[na:na]
at org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:345) ~[na:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_111]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_111]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Caused by: org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.net.ConnectException: Connection refused
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1533) ~[na:na]
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1553) ~[na:na]
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1704) ~[na:na]
at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) ~[na:na]
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124) ~[na:na]
... 19 common frames omitted
Caused by: com.google.protobuf.ServiceException: java.net.ConnectException: Connection refused
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223) ~[na:na]
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287) ~[na:na]
at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:50918) ~[na:na]
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1564) ~[na:na]
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1502) ~[na:na]
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1524) ~[na:na]
... 23 common frames omitted
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_111]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_111]
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) ~[na:na]
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) ~[na:na]
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) ~[na:na]
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupConnection(RpcClientImpl.java:424) ~[na:na]
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:748) ~[na:na]
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:920) ~[na:na]
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:889) ~[na:na]
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1222) ~[na:na]
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213) ~[na:na]
... 28 common frames omitted
2017-01-25 22:23:03,348 ERROR [StandardProcessScheduler Thread-7] o.a.n.c.s.StandardControllerServiceNode Failed to invoke @OnEnabled method of HBase_1_1_2_ClientService[id=d3eaf393-0159-1000-ffff-ffffa95f1940] due to org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions:
Wed Jan 25 22:23:03 UTC 2017, RpcRetryingCaller{globalStartTime=1485382983338, pause=100, retries=1}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.net.ConnectException: Connection refused
我目前仍在处理以下问题:
是否有人设置nifi hdf docker容器将数据存储到hbase hdp docker容器中?
暂无答案!
目前还没有任何答案,快来回答吧!