geospark transform sql函数失败

dzhpxtsq  于 2021-05-27  发布在  Spark
关注(0)|答案(1)|浏览(589)

我正在使用geospark 1.3.1,在这里我试图找到包含在一个圆中的所有地理点,给定一个以米为单位的圆心和半径。要做到这一点,我不想把中心从度转换到米,创建一个圆(使用 ST_Buffer )然后在应用 ST_Contains 与所有地理点连接。请参见下面的sql:

  1. WITH point_data AS (
  2. SELECT
  3. ST_Point(CAST(c.lon as Decimal(24,20)), CAST(c.lat as Decimal(24,20))) as geo_point
  4. FROM point_data_view as c
  5. )
  6. SELECT * FROM point_data as pd
  7. WHERE ST_Contains(ST_Transform(ST_Buffer(ST_Transform(ST_Point(<LON>, <LAT>), 'epsg:4326', 'epsg:3857'), 1000.0), 'epsg:3857', 'epsg:4326'), pd.geo_point) = true

然而,当我按照指南geospark添加依赖到pom文件并创建一个uberjar提交(使用 spark2-submit )我得到以下错误(仅当使用 ST_Transform (功能)

  1. java.lang.NoSuchMethodError: org.hsqldb.DatabaseURL.parseURL(Ljava/lang/String;ZZ)Lorg/hsqldb/persist/HsqlProperties;
  2. at org.hsqldb.jdbc.JDBCDriver.getConnection(Unknown Source)
  3. at org.hsqldb.jdbc.JDBCDataSource.getConnection(Unknown Source)
  4. at org.hsqldb.jdbc.JDBCDataSource.getConnection(Unknown Source)
  5. at org.geotools.referencing.factory.epsg.DirectEpsgFactory.getConnection(DirectEpsgFactory.java:3302)
  6. at org.geotools.referencing.factory.epsg.ThreadedEpsgFactory.createBackingStore(ThreadedEpsgFactory.java:436)
  7. at org.geotools.referencing.factory.DeferredAuthorityFactory.getBackingStore(DeferredAuthorityFactory.java:133)
  8. at org.geotools.referencing.factory.BufferedAuthorityFactory.isAvailable(BufferedAuthorityFactory.java:235)
  9. at org.geotools.referencing.factory.DeferredAuthorityFactory.isAvailable(DeferredAuthorityFactory.java:119)
  10. at org.geotools.factory.FactoryRegistry.isAvailable(FactoryRegistry.java:667)
  11. at org.geotools.factory.FactoryRegistry.isAcceptable(FactoryRegistry.java:501)
  12. at org.geotools.factory.FactoryRegistry.getServiceImplementation(FactoryRegistry.java:437)
  13. at org.geotools.factory.FactoryRegistry.getServiceProvider(FactoryRegistry.java:365)
  14. at org.geotools.factory.FactoryCreator.getServiceProvider(FactoryCreator.java:145)
  15. at org.geotools.referencing.ReferencingFactoryFinder.getAuthorityFactory(ReferencingFactoryFinder.java:220)
  16. at org.geotools.referencing.ReferencingFactoryFinder.getCRSAuthorityFactory(ReferencingFactoryFinder.java:440)
  17. at org.geotools.referencing.factory.epsg.LongitudeFirstFactory.createBackingStore(LongitudeFirstFactory.java:192)
  18. at org.geotools.referencing.factory.DeferredAuthorityFactory.getBackingStore(DeferredAuthorityFactory.java:133)
  19. at org.geotools.referencing.factory.BufferedAuthorityFactory.isAvailable(BufferedAuthorityFactory.java:235)
  20. at org.geotools.referencing.factory.DeferredAuthorityFactory.isAvailable(DeferredAuthorityFactory.java:119)
  21. at org.geotools.factory.FactoryRegistry.isAvailable(FactoryRegistry.java:667)
  22. at org.geotools.factory.FactoryRegistry.isAcceptable(FactoryRegistry.java:501)
  23. at org.geotools.factory.FactoryRegistry$1.filter(FactoryRegistry.java:192)
  24. at javax.imageio.spi.FilterIterator.advance(ServiceRegistry.java:834)
  25. at javax.imageio.spi.FilterIterator.<init>(ServiceRegistry.java:828)
  26. at javax.imageio.spi.ServiceRegistry.getServiceProviders(ServiceRegistry.java:519)
  27. at org.geotools.factory.FactoryRegistry.getServiceProviders(FactoryRegistry.java:197)
  28. at org.geotools.referencing.ReferencingFactoryFinder.getFactories(ReferencingFactoryFinder.java:180)
  29. at org.geotools.referencing.ReferencingFactoryFinder.getCRSAuthorityFactories(ReferencingFactoryFinder.java:455)
  30. at org.geotools.referencing.DefaultAuthorityFactory.getBackingFactory(DefaultAuthorityFactory.java:89)
  31. at org.geotools.referencing.DefaultAuthorityFactory.<init>(DefaultAuthorityFactory.java:69)
  32. at org.geotools.referencing.CRS.getAuthorityFactory(CRS.java:263)
  33. at org.geotools.referencing.CRS.decode(CRS.java:525)
  34. at org.geotools.referencing.CRS.decode(CRS.java:453)
  35. at org.apache.spark.sql.geosparksql.expressions.ST_Transform.eval(Functions.scala:237)

我曾尝试在我的构建中对“org.hsqldb”进行着色和重新定位,但这并没有改变任何事情,而且没有将geospark jar包含在uber jar中,而是将它们作为构建的一部分进行加载 spark2-submit 我也有同样的错误。我真的找不到解决这个问题的方法,只有使用 ST_Transform 方法?
看起来myspark平台有一个旧版本的 org.hsqldb 比地球公园给你的还要多!
我的底纹如下:

  1. <properties>
  2. <geoSparkVersion>1.3.1</geoSparkVersion>
  3. <sparkVersion>2.3.0</sparkVersion>
  4. <clouderaPackage>cloudera2</clouderaPackage>
  5. <scalaVersion>2.11.0</scalaVersion>
  6. <scalaBinaryVersion>2.11</scalaBinaryVersion>
  7. </properties>
  8. <dependencies>
  9. <dependency>
  10. <groupId>org.apache.spark</groupId>
  11. <artifactId>spark-core_${scalaBinaryVersion}</artifactId>
  12. <version>${sparkVersion}.${clouderaPackage}</version>
  13. <scope>provided</scope>
  14. </dependency>
  15. <dependency>
  16. <groupId>org.apache.spark</groupId>
  17. <artifactId>spark-sql_${scalaBinaryVersion}</artifactId>
  18. <version>${sparkVersion}.${clouderaPackage}</version>
  19. <scope>provided</scope>
  20. </dependency>
  21. <dependency>
  22. <groupId>org.datasyslab</groupId>
  23. <artifactId>geospark</artifactId>
  24. <version>${geoSparkVersion}</version>
  25. </dependency>
  26. <dependency>
  27. <groupId>org.datasyslab</groupId>
  28. <artifactId>geospark-sql_2.3</artifactId>
  29. <version>${geoSparkVersion}</version>
  30. </dependency>
  31. </dependencies>
  32. <build>
  33. <plugins>
  34. <plugin>
  35. <groupId>org.apache.maven.plugins</groupId>
  36. <artifactId>maven-shade-plugin</artifactId>
  37. <version>3.2.1</version>
  38. <configuration>
  39. <shadedArtifactAttached>true</shadedArtifactAttached>
  40. <finalName>${artifactId}-${version}-${jarNameWithDependencies}</finalName>
  41. <filters>
  42. <filter>
  43. <artifact>*:*</artifact>
  44. <excludes>
  45. <exclude>META-INF/*.SF</exclude>
  46. <exclude>META-INF/*.DSA</exclude>
  47. <exclude>META-INF/*.RSA</exclude>
  48. </excludes>
  49. </filter>
  50. </filters>
  51. <relocations>
  52. <relocation>
  53. <pattern>org.hsqldb</pattern>
  54. <shadedPattern>shaded.org.hsqldb</shadedPattern>
  55. </relocation>
  56. </relocations>
  57. </configuration>
  58. <executions>
  59. <execution>
  60. <phase>package</phase>
  61. <goals>
  62. <goal>shade</goal>
  63. </goals>
  64. </execution>
  65. </executions>
  66. </plugin>
  67. </plugins>
  68. </build>
hwamh0ep

hwamh0ep1#

看起来像是经典的依赖关系问题,可能是您的ubber jar容器org.hsqldb库的不同版本,您应该尝试从您的依赖关系中排除org.hsqldb.*或对其进行着色处理。我猜你用maven shaded插件做uber jar?如果您这样做了,您可以在这里查看如何排除依赖项:https://maven.apache.org/plugins/maven-shade-plugin/examples/includes-excludes.html

相关问题