无法使用es hadoop在elasticsearch中创建外部表

mklgxw1f  于 2021-06-26  发布在  Hive
关注(0)|答案(0)|浏览(482)

我正在运行一个简单的spark提交作业,例如:

  1. enter code here spark-submit --class com.x.y.z.logan
  2. /home/test/spark/sample.jar
  3. table in jar file
  4. hiveContext.sql("CREATE TABLE IF NOT EXISTS
  5. databasename.tablename(es_column_name STRING) STORED BY
  6. 'org.elasticsearch.hadoop.hive.EsStorageHandler'
  7. TBLPROPERTIES('es.resource' = 'index_name/log','es.mapping.names'
  8. ='tablecoulmname :es_column_name ', 'es.nodes' =
  9. '192.168.x.1y:9200','es.input.json' = 'false',
  10. 'es.index.read.missing.as.empty' = 'yes' ,'es.index.auto.create' =
  11. 'yes') ")
  12. hiveContext.sql("INSERT INTO TABLE test.incompleterun SELECT
  13. s.streamname FROM incomplete s");
  14. **ERROR**
  15. client token: N/A
  16. diagnostics: User class threw exception: java.lang.RuntimeException:
  17. java.lang.RuntimeException: class
  18. org.elasticsearch.hadoop.mr.EsOutputFormat$EsOutputCommitter not
  19. org.apache.hadoop.mapred.OutputCommitter
  20. ApplicationMaster host: 192.168.x.y
  21. ApplicationMaster RPC port: 0
  22. queue: root.users.test
  23. start time: 1485286033939
  24. final status: FAILED
  25. tracking URL: some URL
  26. user: test
  27. Exception in thread "main" org.apache.spark.SparkException:
  28. Application application_1485258812942_0008 finished with failed status
  29. at org.apache.spark.deploy.yarn.Client.run(Client.scala:1035)
  30. at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1082)
  31. at org.apache.spark.deploy.yarn.Client.main(Client.scala)
  32. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

当我们在配置单元接口中创建外部表并将数据从配置单元表加载到外部表时,es hadoop工作正常。当我们在jar中包含相同的查询时,它不起作用。当我们创建普通的配置单元表时,相同的jar文件可以正常工作。这里的问题是当我们在jar文件中包含外部表时出现了下面的错误,有人能帮我解决这个问题吗?

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题