jsonpath 2.3.0与hadoop2.7环境json-smart1.2.0.jar冲突

at0kjp5o  于 2021-06-24  发布在  Pig
关注(0)|答案(2)|浏览(508)

在hdfs位置中输入json
需要解析json并聚合结果
为此，我使用了pig udf，它使用json路径库
在hadoop2.7环境jar中：json-smart1.2、json-path1.2是硬绑定的
每当我执行pig mapreduce时，它将我抛出以下异常

  1. java.lang.NoSuchFieldError: defaultReader
  2. at com.jayway.jsonpath.spi.json.JsonSmartJsonProvider.<init>(JsonSmartJsonProvider.java:39)
  3. at com.jayway.jsonpath.internal.DefaultsImpl.jsonProvider(DefaultsImpl.java:21)
  4. at com.jayway.jsonpath.Configuration.defaultConfiguration(Configuration.java:174)
  5. at com.jayway.jsonpath.internal.JsonContext.<init>(JsonContext.java:52)
  6. at com.jayway.jsonpath.JsonPath.parse(JsonPath.java:596)

为了解决这个问题，尝试了以下几种选择
选项1：尝试设置注册json-smart2.3.jar&json-path2.3.0.jar，但没有很好的结果（因为它引用的jar是json-path1.2.jar）
选项2：将我的模块依赖关系降级到json-path1.2.jar没有结果
选项3：使用自定义类加载器尝试加载json-path2.3.0jar的jar，它加载的类出现org.slf4j绑定问题
标识了多个绑定路径，但sun.misc classloader出现问题

  1. Failed to instantiate SLF4J LoggerFactory
  2. Reported exception:
  3. java.lang.NullPointerException
  4.     at sun.net.util.URLUtil.urlNoFragString(URLUtil.java:50)
  5.     at sun.misc.URLClassPath.getLoader(URLClassPath.java:485)
  6.     at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:457)
  7.     at sun.misc.URLClassPath.access$100(URLClassPath.java:64)
  8.     at sun.misc.URLClassPath$1.next(URLClassPath.java:239)
  9.     at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:250)
  10.     at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601)
  11.     at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599)
  12.     at java.security.AccessController.doPrivileged(Native Method)
  13.     at java.net.URLClassLoader$3.next(URLClassLoader.java:598)
  14.     at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623)
  15.     at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
  16.     at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
  17.     at org.slf4j.LoggerFactory.findPossibleStaticLoggerBinderPathSet(LoggerFactory.java:238)
  18.     at org.slf4j.LoggerFactory.bind(LoggerFactory.java:138)
  19.     at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
  20.     at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
  21.     at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
  22.     at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:304)
  23.     at com.jayway.jsonpath.internal.JsonContext.<clinit>(JsonContext.java:41)
  24.     at com.jayway.jsonpath.internal.ParseContextImpl.parse(ParseContextImpl.java:38)
  25.     at com.jayway.jsonpath.JsonPath.parse(JsonPath.java:599)
  26.     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  27.     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  28.     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  29.     at java.lang.reflect.Method.invoke(Method.java:498)
  30.     at com.optum.pdm.ReferenceDataUpdate.addURL(ReferenceDataUpdate.java:112)
  31.     at com.optum.pdm.ReferenceDataUpdate.main(ReferenceDataUpdate.java:124)
  32. Exception in thread "main" java.lang.reflect.InvocationTargetException
  33.     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  34.     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  35.     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  36.     at java.lang.reflect.Method.invoke(Method.java:498)
  37.     at com.optum.pdm.ReferenceDataUpdate.addURL(ReferenceDataUpdate.java:112)
  38.     at com.optum.pdm.ReferenceDataUpdate.main(ReferenceDataUpdate.java:124)
  39. Caused by: java.lang.ExceptionInInitializerError
  40.     at com.jayway.jsonpath.internal.ParseContextImpl.parse(ParseContextImpl.java:38)
  41.     at com.jayway.jsonpath.JsonPath.parse(JsonPath.java:599)
  42.     ... 6 more
  43. Caused by: java.lang.IllegalStateException: Unexpected initialization failure
  44.     at org.slf4j.LoggerFactory.bind(LoggerFactory.java:167)
  45.     at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
  46.     at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
  47.     at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
  48.     at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:304)
  49.     at com.jayway.jsonpath.internal.JsonContext.<clinit>(JsonContext.java:41)
  50.     ... 8 more
  51. Caused by: java.lang.NullPointerException
  52.     at sun.net.util.URLUtil.urlNoFragString(URLUtil.java:50)
  53.     at sun.misc.URLClassPath.getLoader(URLClassPath.java:485)
  54.     at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:457)
  55.     at sun.misc.URLClassPath.access$100(URLClassPath.java:64)
  56.     at sun.misc.URLClassPath$1.next(URLClassPath.java:239)
  57.     at sun.misc.URLClassPath$1.hasMoreElements(URLClassPath.java:250)
  58.     at java.net.URLClassLoader$3$1.run(URLClassLoader.java:601)
  59.     at java.net.URLClassLoader$3$1.run(URLClassLoader.java:599)
  60.     at java.security.AccessController.doPrivileged(Native Method)
  61.     at java.net.URLClassLoader$3.next(URLClassLoader.java:598)
  62.     at java.net.URLClassLoader$3.hasMoreElements(URLClassLoader.java:623)
  63.     at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
  64.     at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
  65.     at org.slf4j.LoggerFactory.findPossibleStaticLoggerBinderPathSet(LoggerFactory.java:238)
  66.     at org.slf4j.LoggerFactory.bind(LoggerFactory.java:138)
  67.     ... 13 more

有人能建议我解决这个问题吗？有人能找到一个stackoverflow链接来说明weblogic，而不是一个同样适用于hadoop2.7的通用解决方案（json解析器-java.lang.nosuchfielderror:defaultreader）

JavaJSONhadoop2apache-pigJsonPath

2条答案

按热度按时间
pn9klfpd

pn9klfpd1#

提供自定义实现来加载/解析json，这样我们就可以避免使用hadoop/lib的json-smart-2.x/1.x

  1. public static void changeJsonPathConfig() {
  3.     if (!configChanged) {
  5.         Configuration.setDefaults(new Configuration.Defaults() {
  7.             private final JsonProvider jsonProvider = new GsonJsonProvider(
  9.                     new GsonBuilder().serializeNulls().create());
  11.             private final MappingProvider mappingProvider = new GsonMappingProvider();
  13.             @Override
  15.             public JsonProvider jsonProvider() {
  17.                 return jsonProvider;
  19.             }
  21.             @Override
  23.             public MappingProvider mappingProvider() {
  25.                 return mappingProvider;
  27.             }
  29.             @Override
  31.             public Set<Option> options() {
  33.                 return EnumSet.noneOf(Option.class);
  35.             }
  37.         });
  39.         configChanged = true;
  41.     }
  43. }
展开查看全部
赞(0）分享  回复(0）举报  2021-06-24
qpgpyjmq

qpgpyjmq2#

由于hadoop环境（pig、hdfs、hive等）使用的是json-path-2.3.0，它更好的用户Map器逻辑应该使用另一个版本“jsonpath-1.0.jar”来解决这个问题

赞(0）分享  回复(0）举报  2021-06-24
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com