使用oozie将hdfs文件Map到hbase

7uhlpewt  于 2021-06-03  发布在  Hadoop
关注(0)|答案(1)|浏览(321)

我正在尝试从oozie启动一个shell作业,在那里我将文件从hdfsMap到hbase
shell脚本可以从命令行工作,但不能从oozie工作
shell脚本:

HADOOP_CLASSPATH=`/usr/bin/hbase classpath` /usr/bin/hadoop jar /usr/lib/hbase/hbase-server.jar importtsv -Dimporttsv.separator=, -Dimporttsv.columns=ORIGINAL:FIELD1,ORIGINAL:FIELD2,ORIGINAL:FIELD3  HBASE_Table /user/U0H8048/file

错误是:

SLF4J: Class path contains multiple SLF4J bindings.
  SLF4J: Found binding in [jar:file:/cloudera/opt/cloudera/parcels/CDH-5.0.2-    1.cdh5.0.2.p0.13/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in [jar:file:/cloudera/yarn/nm/filecache/2282/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in [jar:file:/cloudera/yarn/nm/filecache/2288/slf4j-simple-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "main" java.lang.reflect.InvocationTargetException
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:54)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
  Caused by: java.io.FileNotFoundException: File does not exist: hdfs://salcls0226.com:8020/cloudera/opt/cloudera/parcels/CDH-5.0.2-1.cdh5.0.2.p0.13/lib/hbase/lib/hbase-server-0.96.1.1-cdh5.0.2.jar
  at    org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1128)
  at    org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1120)
      ...

有人知道可能的解决办法吗?
如何指定 hbase-server-0.96.1.1-cdh5.0.2.jar 文件位于本地路径而不在hdfs上?
谢谢

pgx2nnw8

pgx2nnw81#

我解决了这个问题。
它需要添加行 oozie.use.system.libpath=true 当工作流启动时,由oozie读取job.properties文件。
您必须将所有需要的文件添加到hdfs路径:/user/oozie/share/lib中。通常主库文件已经存在。

相关问题