gobblin kafka to hdfs pull job错误

4nkexdtk  于 2021-06-02  发布在  Hadoop
关注(0)|答案(1)|浏览(436)

我正在尝试使用gobblin从kafka到hdfs的数据。
gobblin版本(使用命令从github源代码编译) sudo ./gradlew clean build -PuseHadoop2 -PhadoopVersion=2.7.1 -x test ):

  1. 0.6.2-546-g431188b

hadoop版本:

  1. Hadoop 2.7.1.2.4.2.0-258
  2. Subversion git@github.com:hortonworks/hadoop.git -r 13debf893a605e8a88df18a7d8d214f571e05289
  3. Compiled by jenkins on 2016-04-24T16:02Z
  4. Compiled with protoc 2.5.0
  5. From source with checksum 2a2d95f05ec6c3ac547ed58cab713ac
  6. This command was run using /usr/hdp/2.4.2.0-258/hadoop/hadoop-common-2.7.1.2.4.2.0-258.jar

gobblin作业:

  1. job.name=GobblinKafkaQuickStart
  2. job.group=GobblinKafka
  3. job.description=Gobblin quick start job for Kafka
  4. job.lock.enabled=false
  5. job.schedule=0 0/2 * * * ?
  6. kafka.brokers=hd-mgt03:6667,hd-mgt02:6667,hd-mgt04:6667
  7. source.class=gobblin.source.extractor.extract.kafka.KafkaSimpleSource
  8. extract.namespace=gobblin.extract.kafka
  9. writer.builder.class=gobblin.writer.AvroHdfsDataWriter
  10. writer.file.path.type=tablename
  11. writer.destination.type=HDFS
  12. writer.output.format=AVRO
  13. data.publisher.type=gobblin.publisher.BaseDataPublisher
  14. mr.job.max.mappers=1
  15. metrics.reporting.file.enabled=true
  16. metrics.log.dir=/gobblin-kafka/metrics
  17. metrics.reporting.file.suffix=txt
  18. bootstrap.with.offset=earliest
  19. fs.uri=hdfs://hdfs:8020
  20. writer.fs.uri=hdfs://hdfs:8020
  21. state.store.fs.uri=hdfs://hdfs:8020
  22. mr.job.root.dir=/kafka/working
  23. state.store.dir=/kafka/state-store
  24. task.data.root.dir=/kafka/task-data
  25. data.publisher.final.dir=/kafka/job-output

我想逃跑 gobblin-mapreduce.shgobblin-dist/bin 文件夹,但获取错误:

  1. Exception in thread "main" gobblin.runtime.JobException: Job job_GobblinKafkaQuickStart_1464962113982 failed
  2. at gobblin.runtime.AbstractJobLauncher.launchJob(AbstractJobLauncher.java:363)
  3. at gobblin.runtime.mapreduce.CliMRJobLauncher.launchJob(CliMRJobLauncher.java:84)
  4. at gobblin.runtime.mapreduce.CliMRJobLauncher.run(CliMRJobLauncher.java:61)
  5. at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
  6. at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
  7. at gobblin.runtime.mapreduce.CliMRJobLauncher.main(CliMRJobLauncher.java:106)
  8. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  9. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  10. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  11. at java.lang.reflect.Method.invoke(Method.java:497)
  12. at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
  13. at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

日志文件包含错误:

  1. 2016-06-03 16:55:17 MSK ERROR [main] gobblin.runtime.AbstractJobLauncher 321 - Failed to launch and run job job_GobblinKafkaQuickStart_1464962113982: java.lang.NoSuchFieldError: DEFAULT_MR_AM_ADMIN_USER_ENV
  2. java.lang.NoSuchFieldError: DEFAULT_MR_AM_ADMIN_USER_ENV
  3. at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:470)
  4. at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:285)
  5. at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240)
  6. at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
  7. at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
  8. at java.security.AccessController.doPrivileged(Native Method)
  9. at javax.security.auth.Subject.doAs(Subject.java:422)
  10. at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
  11. at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
  12. at gobblin.runtime.mapreduce.MRJobLauncher.runWorkUnits(MRJobLauncher.java:198)
  13. at gobblin.runtime.AbstractJobLauncher.launchJob(AbstractJobLauncher.java:296)
  14. at gobblin.runtime.mapreduce.CliMRJobLauncher.launchJob(CliMRJobLauncher.java:84)
  15. at gobblin.runtime.mapreduce.CliMRJobLauncher.run(CliMRJobLauncher.java:61)
  16. at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
  17. at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
  18. at gobblin.runtime.mapreduce.CliMRJobLauncher.main(CliMRJobLauncher.java:106)
  19. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  20. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  21. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  22. at java.lang.reflect.Method.invoke(Method.java:497)
  23. at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
  24. at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

这个错误的原因是什么?我该怎么修?

fslejnso

fslejnso1#

从你的错误我可以看出可能是jar的问题。通常,这个错误( java.lang.NoSuchFieldError: DEFAULT_MR_AM_ADMIN_USER_ENV )是由jar冲突引起的。您可以检查类路径以查看是否存在任何版本冲突。

相关问题