在java代码中连接到cloudera hdfs,得到不满意的链接错误

l7mqbcuq  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(371)

因此,我正在尝试编写可以连接到在amazonec2示例上运行的cloudera(cdh4.4)集群上的hdfs的代码。我有所有需要打开的端口,除了代码之外,我几乎可以通过其他方式(webui、ssh等)访问集群。任何时候,我都会得到:

  1. 15:07:35.115 [main] DEBUG o.a.h.m.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)], about=, type=DEFAULT, always=false, sampleName=Ops)
  2. 15:07:35.125 [main] DEBUG o.a.h.m.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)], about=, type=DEFAULT, always=false, sampleName=Ops)
  3. 15:07:35.127 [main] DEBUG o.a.h.m.impl.MetricsSystemImpl - UgiMetrics, User and group related metrics
  4. 15:07:35.605 [main] DEBUG o.a.h.s.a.util.KerberosName - Kerberos krb5 configuration not found, setting default realm to empty
  5. 15:07:35.651 [main] DEBUG org.apache.hadoop.security.Groups - Creating new Groups object
  6. 15:07:35.654 [main] DEBUG o.a.hadoop.util.NativeCodeLoader - Trying to load the custom-built native-hadoop library...
  7. 15:07:35.654 [main] DEBUG o.a.hadoop.util.NativeCodeLoader - Loaded the native-hadoop library
  8. java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
  9. at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
  10. at org.apache.hadoop.security.Groups.<init>(Groups.java:55)
  11. at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:182)
  12. at org.apache.hadoop.security.UserGroupInformation.initUGI(UserGroupInformation.java:252)
  13. at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:223)
  14. at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:214)
  15. at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:277)
  16. at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:668)
  17. at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:573)
  18. at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2464)
  19. at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2456)
  20. at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2323)
  21. at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:351)
  22. at org.skye.hadoop.stores.HdfsArchiveStore.initialize(HdfsArchiveStore.java:44)
  23. at org.skye.hadoop.HdfsArchiveStoreTest.testHdfsConfig(HdfsArchiveStoreTest.java:15)
  24. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  25. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  26. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  27. at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
  28. at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
  29. at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
  30. at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
  31. at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
  32. at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
  33. at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
  34. at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
  35. at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
  36. at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
  37. at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
  38. at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
  39. at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
  40. at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
  41. at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:76)
  42. at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:195)
  43. at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:63)
  44. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  45. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  46. at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
  47. Caused by: java.lang.reflect.InvocationTargetException
  48. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  49. at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
  50. at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  51. at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
  52. at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:129)
  53. ... 40 more
  54. Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.security.JniBasedUnixGroupsMapping.anchorNative()V
  55. at org.apache.hadoop.security.JniBasedUnixGroupsMapping.anchorNative(Native Method)
  56. at org.apache.hadoop.security.JniBasedUnixGroupsMapping.<clinit>(JniBasedUnixGroupsMapping.java:49)
  57. at org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.<init>(JniBasedUnixGroupsMappingWithFallback.java:38)
  58. ... 45 more

基本上我找不到任何地方可以解释从哪里开始寻找导致这个错误的原因。有人吗?

vhmi4jdf

vhmi4jdf1#

“15:07:35.654[main]调试o.a.hadoop.util.nativecodeloader-已加载本机hadoop库”
可能您加载的本机库不正确。在“java.library.path”中检查它们。

相关问题