talend hivedb连接需要cloudera serde

wgx48brx  于 2021-05-29  发布在  Hadoop
关注(0)|答案(1)|浏览(446)

我想把泰伦德开放工作室和Hive连接起来。
在hive中，我有一个带有自定义字段的表（来自clouderatwitter示例）。
打开studio find table没有问题，但如果尝试检索架构，会出现以下错误：

在talend日志中我得到：

!ENTRY org.talend.platform.logging 4 0 2015-09-21 10:49:12.375
!MESSAGE 2015-09-21 10:49:12,372 ERROR org.talend.commons.exception.CommonExceptionHandler  - java.lang.reflect.InvocationTargetException

!STACK 0
java.sql.SQLException: java.lang.reflect.InvocationTargetException
    at org.talend.metadata.managment.hive.EmbeddedHiveDataBaseMetadata.getColumns(EmbeddedHiveDataBaseMetadata.java:401)
    at org.talend.core.model.metadata.builder.database.manager.ExtractManager.getColumnsResultSet(ExtractManager.java:844)
    at org.talend.core.model.metadata.builder.database.manager.ExtractManager.extractColumns(ExtractManager.java:641)
    at org.talend.core.model.metadata.builder.database.manager.ExtractManager.returnMetadataColumnsFormTable(ExtractManager.java:521)
    at org.talend.core.model.metadata.builder.database.ExtractMetaDataFromDataBase.returnMetadataColumnsFormTable(ExtractMetaDataFromDataBase.java:224)
    at org.talend.repository.ui.wizards.metadata.table.database.DatabaseTableForm.pressRetreiveSchemaButton(DatabaseTableForm.java:1150)
    at org.talend.repository.ui.wizards.metadata.table.database.DatabaseTableForm.access$13(DatabaseTableForm.java:1121)
    at org.talend.repository.ui.wizards.metadata.table.database.DatabaseTableForm$4.widgetSelected(DatabaseTableForm.java:795)
    at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:248)
    at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
    at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4454)
    at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1388)
    at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:3799)
    at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3409)
    at org.eclipse.jface.window.Window.runEventLoop(Window.java:832)
    at org.eclipse.jface.window.Window.open(Window.java:808)
    at org.talend.repository.metadata.ui.actions.metadata.AbstractCreateTableAction.handleWizard(AbstractCreateTableAction.java:139)
    at org.talend.repository.metadata.ui.actions.metadata.AbstractCreateTableAction$1$1.run(AbstractCreateTableAction.java:1088)
    at org.talend.repository.RepositoryWorkUnit.executeRun(RepositoryWorkUnit.java:93)
    at org.talend.core.repository.model.AbstractRepositoryFactory.executeRepositoryWorkUnit(AbstractRepositoryFactory.java:256)
    at org.talend.repository.localprovider.model.LocalRepositoryFactory.executeRepositoryWorkUnit(LocalRepositoryFactory.java:3227)
    at org.talend.core.repository.model.ProxyRepositoryFactory.executeRepositoryWorkUnit(ProxyRepositoryFactory.java:1996)
    at org.talend.repository.metadata.ui.actions.metadata.AbstractCreateTableAction$1.runInUIThread(AbstractCreateTableAction.java:1110)
    at org.eclipse.ui.progress.UIJob$1.run(UIJob.java:97)
    at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:35)
    at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:136)
    at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:3774)
    at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3412)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run(PartRenderingEngine.java:1151)
    at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1032)
    at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:148)
    at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:636)
    at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
    at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:579)
    at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:150)
    at org.talend.rcp.intro.Application.start(Application.java:183)
    at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
    at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
    at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
    at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:380)
    at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:235)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:648)
    at org.eclipse.equinox.launcher.Main.basicRun(Main.java:603)
    at org.eclipse.equinox.launcher.Main.run(Main.java:1465)
    at org.eclipse.equinox.launcher.Main.main(Main.java:1438)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.talend.metadata.managment.hive.EmbeddedHiveDataBaseMetadata.getColumns(EmbeddedHiveDataBaseMetadata.java:370)
    ... 49 more
Caused by: java.lang.RuntimeException: MetaException(message:java.lang.ClassNotFoundException Class com.cloudera.hive.serde.JSONSerDe not found)
    at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:276)
    at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:256)
    at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:595)
    at org.apache.hadoop.hive.ql.metadata.Table.getAllCols(Table.java:612)
    ... 54 more
Caused by: MetaException(message:java.lang.ClassNotFoundException Class com.cloudera.hive.serde.JSONSerDe not found)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:385)
    at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:274)
    ... 57 more

最重要的信息是

Caused by: java.lang.RuntimeException: MetaException(message:java.lang.ClassNotFoundException Class com.cloudera.hive.serde.JSONSerDe not found)
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:276)
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:256)
at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:595)
at org.apache.hadoop.hive.ql.metadata.Table.getAllCols(Table.java:612)
... 54 more
Caused by: MetaException(message:java.lang.ClassNotFoundException Class com.cloudera.hive.serde.JSONSerDe not found)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:385)
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:274)
... 57 more

此消息显示talend open studio for big data找不到serde。
但我将hiveserede放在所有hadoop集群节点上，informatica powercenter可以从hive获取模式信息。
另外，当我在节点或色调上使用hivecli在配置单元上运行查询时，它不会出现任何问题和额外的配置，比如
添加jar
如何让talend open studio与我的Hive表一起工作？
hadoop:cdh5.4.3
talend开放工作室6.0.1
Hive1.1.0+cdh5.4.3+151

hadoopHivetalendbigdata

1条答案

按热度按时间
xv8emn3q

xv8emn3q1#

我解决了这个问题。默认情况下，talend open studio尝试直接使用配置单元元存储。
所谓的嵌入式连接(metastore端口为9093）
但在连接设置中，我看到端口10000将我指向hiveserver2（thrift）。
切换到独立连接后，中的开始工作。

赞(0）分享  回复(0）举报  2021-05-30
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com