java—无法通过ApacheIceberg将数据写入表

mm5n2pyu  于 2021-07-11  发布在  Java
关注(0)|答案(1)|浏览(509)

我试图用ApacheIceberg0.9.1将简单的数据写入表中，但是错误消息显示。我想用hadoop直接积垢数据。我创建了一个hadoop表，并尝试从表中读取数据。之后，我尝试将数据写入表中。我准备了一个包含一行的json文件。我的代码已经读取了json对象，并排列了数据的顺序，但是最后一步写入数据总是错误的。我已经更改了依赖项包的一些版本，但是显示了另一个错误消息。软件包的版本有问题吗。请帮帮我。
这是我的源代码：

  1. public class IcebergTest {
  3.     public static void main(String[] args) {
  4.         testWithoutCatalog();
  5.         readDataWithouCatalog();
  6.         writeDataWithoutCatalog();
  8.     }
  10.     public static void testWithoutCatalog() {
  12.         Schema bookSchema = new Schema(optional(1, "title", Types.StringType.get()),
  13.                 optional(2, "price", Types.LongType.get()), 
  14.                 optional(3, "author", Types.StringType.get()),               
  15.                 optional(4, "genre", Types.StringType.get()));
  16.         PartitionSpec bookspec = PartitionSpec.builderFor(bookSchema).identity("title").build();
  18.         Configuration conf = new Configuration();
  20.         String warehousePath = "hdfs://hadoop01:9000/warehouse_path/xgfying/books3";
  22.         HadoopTables tables = new HadoopTables(conf);
  23.         Table table = tables.create(bookSchema, bookspec, warehousePath);
  24.     }
  26.     public static void readDataWithouCatalog(){
  27.         .......
  28.     }
  30.     public static void writeDataWithoutCatalog(){
  31.         SparkSession spark = SparkSession.builder().master("local[2]").getOrCreate();
  32.         Dataset<Row> df = spark.read().json("src/test/data/books3.json");       
  33.         System.out.println(" this is the writing data : "+df.select("title","price","author","genre")
  34.                                                             .first().toString());
  35.         df.select("title","price","author","genre")
  36.           .write().format("iceberg").mode("append")
  37.           .save("hdfs://hadoop01:9000/warehouse_path/xgfying/books3");
  38.         // System.out.println(df.write().format("iceberg").mode("append").toString());
  39.     }
  41. }

以下是错误消息：

  1. Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
  2. 20/11/18 15:51:36 INFO SparkContext: Running Spark version 2.4.5
  3. .......
  4. file:///C:/tmp/icebergtest1/src/test/data/books3.json, range: 0-75, partition values: [empty row]
  5. 20/11/18 15:51:52 ERROR Utils: Aborting task
  6. java.lang.ExceptionInInitializerError
  7.         at org.apache.iceberg.parquet.Parquet$WriteBuilder.build(Parquet.java:232)
  8.         at org.apache.iceberg.spark.source.SparkAppenderFactory.newAppender(SparkAppenderFactory.java:61)
  9.         at org.apache.iceberg.spark.source.BaseWriter.openCurrent(BaseWriter.java:105)
  10.         at org.apache.iceberg.spark.source.PartitionedWriter.write(PartitionedWriter.java:63)
  11.         at org.apache.iceberg.spark.source.Writer$Partitioned24Writer.write(Writer.java:271)
  12.         at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$$anonfun$run$3.apply(WriteToDataSourceV2Exec.scala:118)
  13.         at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$$anonfun$run$3.apply(WriteToDataSourceV2Exec.scala:116)
  14.         at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1394)
  15.         at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:146)
  16.         at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec$$anonfun$doExecute$2.apply(WriteToDataSourceV2Exec.scala:67)
  17.         at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec$$anonfun$doExecute$2.apply(WriteToDataSourceV2Exec.scala:66)
  18.         at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  19.         at org.apache.spark.scheduler.Task.run(Task.scala:123)
  20.         at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
  21.         at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
  22.         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
  23.         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  24.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  25.         at java.lang.Thread.run(Thread.java:748)
  26. Caused by: java.lang.RuntimeException: Cannot find constructor for interface org.apache.parquet.column.page.PageWriteStore       
  27.         Missing org.apache.parquet.hadoop.ColumnChunkPageWriteStore(org.apache.parquet.hadoop.CodecFactory$BytesCompressor,org.apache.parquet.schema.MessageType,org.apache.parquet.bytes.ByteBufferAllocator,int) [java.lang.NoSuchMethodException: org.apache.parquet.hadoop.ColumnChunkPageWriteStore.<init>(org.apache.parquet.hadoop.CodecFactory$BytesCompressor, org.apache.parquet.schema.MessageType, org.apache.parquet.bytes.ByteBufferAllocator, int)]
  28.         at org.apache.iceberg.common.DynConstructors$Builder.build(DynConstructors.java:235)
  29.         at org.apache.iceberg.parquet.ParquetWriter.<clinit>(ParquetWriter.java:55)
  30.         ... 19 more
  31. 20/11/18 15:51:52 ERROR DataWritingSparkTask: Aborting commit for partition 0 (task 2, attempt 0, stage 2.0)
  32. 20/11/18 15:51:52 ERROR DataWritingSparkTask: Aborted commit for partition 0 (task 2, attempt 0, stage 2.0)

这是我的pom.xml：

  1. <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  2.   <modelVersion>4.0.0</modelVersion>
  4.   <groupId>icebergtest</groupId>
  5.   <artifactId>icebergtest1</artifactId>
  6.   <version>0.0.1-SNAPSHOT</version>
  7.   <packaging>jar</packaging>
  9.   <name>icebergtest1</name>
  10.   <url>http://maven.apache.org</url>
  12.   <properties>
  13.     <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  14.         <iceberg.version>0.9.1</iceberg.version>
  15.         <hadoop.version>2.7.0</hadoop.version>
  16.         <maven.compiler.source>1.8</maven.compiler.source>
  17.         <maven.compiler.target>1.8</maven.compiler.target>
  18.   </properties>
  20.   <dependencies>
  21.     <dependency>
  22.       <groupId>junit</groupId>
  23.       <artifactId>junit</artifactId>
  24.       <version>3.8.1</version>
  25.       <scope>test</scope>
  26.     </dependency>
  28.           <!-- org.apache.hadoop BEGIN-->
  29.       <dependency>
  30.           <groupId>org.apache.hadoop</groupId>
  31.           <artifactId>hadoop-common</artifactId>
  32.           <version>${hadoop.version}</version>
  33.       </dependency>
  35.       <dependency>
  36.           <groupId>org.apache.hadoop</groupId>
  37.           <artifactId>hadoop-hdfs</artifactId>
  38.           <version>${hadoop.version}</version>
  39.       </dependency>
  41.       <dependency>
  42.           <groupId>org.apache.hadoop</groupId>
  43.           <artifactId>hadoop-client</artifactId>
  44.           <version>${hadoop.version}</version>
  46.     <!--将netty包排除-->
  47.       <exclusions>
  48.         <exclusion>
  49.           <groupId>io.netty</groupId>
  50.           <artifactId>netty</artifactId>
  51.         </exclusion>
  52.       </exclusions>
  54.       </dependency>
  56.       <!--解决io.netty.buffer.PooledByteBufAllocator.defaultNumHeapArena()I异常，-->
  57.       <dependency>
  58.           <groupId>io.netty</groupId>
  59.           <artifactId>netty-all</artifactId>
  60.           <version>4.1.18.Final</version>
  61.       </dependency>
  63.       <dependency>
  64.           <groupId>org.apache.hadoop</groupId>
  65.           <artifactId>hadoop-mapreduce-client-core</artifactId>
  66.           <version>${hadoop.version}</version>
  67.       </dependency>
  69.           <dependency>
  70.           <groupId>org.apache.hadoop</groupId>
  71.           <artifactId>hadoop-auth</artifactId>
  72.           <version>${hadoop.version}</version>
  73.       </dependency>
  75.           <dependency>
  76.           <groupId>org.apache.hadoop</groupId>
  77.           <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
  78.           <version>${hadoop.version}</version>
  79.       </dependency>
  80.       <!-- org.apache.hadoop END-->
  82.       <!-- org.apache.iceberg BEGIN-->
  83.       <dependency>
  84.         <groupId>org.apache.iceberg</groupId>
  85.         <artifactId>iceberg-core</artifactId>
  86.         <version>${iceberg.version}</version>
  87.       </dependency>
  89.       <dependency>
  90.           <groupId>org.apache.iceberg</groupId>
  91.           <artifactId>iceberg-api</artifactId>
  92.           <version>${iceberg.version}</version>
  93.       </dependency>
  95.         <dependency>
  96.           <groupId>org.apache.iceberg</groupId>
  97.           <artifactId>iceberg-parquet</artifactId>
  98.           <version>${iceberg.version}</version>
  99.       </dependency>
  101.         <dependency>
  102.           <groupId>org.apache.iceberg</groupId>
  103.           <artifactId>iceberg-common</artifactId>
  104.           <version>${iceberg.version}</version>
  105.       </dependency>
  107.         <dependency>
  108.           <groupId>org.apache.iceberg</groupId>
  109.           <artifactId>iceberg-orc</artifactId>
  110.           <version>${iceberg.version}</version>
  111.       </dependency>
  113.         <dependency>
  114.           <groupId>org.apache.iceberg</groupId>
  115.           <artifactId>iceberg-data</artifactId>
  116.           <version>${iceberg.version}</version>
  117.       </dependency>
  119.         <dependency>
  120.           <groupId>org.apache.iceberg</groupId>
  121.           <artifactId>iceberg-hive</artifactId>
  122.           <version>${iceberg.version}</version>
  123.       </dependency>
  125.         <dependency>
  126.           <groupId>org.apache.iceberg</groupId>
  127.           <artifactId>iceberg-arrow</artifactId>
  128.           <version>${iceberg.version}</version>
  129.       </dependency>
  131.         <dependency>
  132.           <groupId>org.apache.iceberg</groupId>
  133.           <artifactId>iceberg-spark</artifactId>
  134.           <version>${iceberg.version}</version>
  135.       </dependency>
  137.         <dependency>
  138.           <groupId>org.apache.iceberg</groupId>
  139.           <artifactId>iceberg-bundled-guava</artifactId>
  140.           <version>${iceberg.version}</version>
  141.       </dependency>
  143.         <dependency>
  144.           <groupId>org.apache.iceberg</groupId>
  145.           <artifactId>iceberg-spark-runtime</artifactId>
  146.           <version>${iceberg.version}</version>
  147.       </dependency>
  149.         <dependency>
  150.           <groupId>org.apache.iceberg</groupId>
  151.           <artifactId>iceberg-spark2</artifactId>
  152.           <version>${iceberg.version}</version>
  153.       </dependency>
  155.         <dependency>
  156.           <groupId>org.apache.iceberg</groupId>
  157.           <artifactId>iceberg-flink</artifactId>
  158.           <version>${iceberg.version}</version>
  159.       </dependency>
  161.         <dependency>
  162.           <groupId>org.apache.iceberg</groupId>
  163.           <artifactId>iceberg-pig</artifactId>
  164.           <version>${iceberg.version}</version>
  165.       </dependency>
  167.       <dependency>
  168.           <groupId>org.apache.iceberg</groupId>
  169.           <artifactId>iceberg-mr</artifactId>
  170.           <version>${iceberg.version}</version>
  171.       </dependency>
  172.       <!-- org.apache.iceberg END-->
  174.         <dependency>
  175.             <groupId>org.apache.spark</groupId>
  176.             <artifactId>spark-sql_2.11</artifactId>
  177.             <version>2.4.5</version>
  178.         </dependency>
  179.         <dependency>
  180.             <groupId>org.apache.spark</groupId>
  181.             <artifactId>spark-core_2.11</artifactId>
  182.             <version>2.4.5</version>
  183.         </dependency>
  185.         <dependency>
  186.             <groupId>org.apache.spark</groupId>
  187.             <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
  188.             <version>2.4.5</version>
  189.         </dependency>
  190.         <dependency>
  191.             <groupId>org.apache.spark</groupId>
  192.             <artifactId>spark-mllib_2.11</artifactId>
  193.             <version>2.4.5</version>
  194.             <exclusions>
  195.                 <exclusion>
  196.                     <groupId>org.codehaus.janino</groupId>
  197.                     <artifactId>commons-compiler</artifactId>
  198.                 </exclusion>
  199.             </exclusions>
  200.         </dependency>
  202.         <dependency>
  203.             <groupId>org.scala-lang</groupId>
  204.             <artifactId>scala-library</artifactId>
  205.             <version>2.11.0</version>
  206.         </dependency>
  207.         <dependency>
  208.             <groupId>org.scala-lang</groupId>
  209.             <artifactId>scala-compiler</artifactId>
  210.             <version>2.11.0</version>
  211.         </dependency>
  212.         <dependency>
  213.             <groupId>org.scala-lang</groupId>
  214.             <artifactId>scala-reflect</artifactId>
  215.             <version>2.11.0</version>
  216.         </dependency>
  218.         <!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind -->
  220.         <dependency>
  221.             <groupId>com.fasterxml.jackson.core</groupId>
  222.             <artifactId>jackson-core</artifactId>
  223.             <!--<version>2.7.9</version>-->
  224.             <version>2.6.6</version>
  225.         </dependency>
  226.         <dependency>
  227.             <groupId>com.fasterxml.jackson.core</groupId>
  228.             <artifactId>jackson-databind</artifactId>
  229.             <!--<version>2.7.9.4</version>-->
  230.             <version>2.6.5</version>
  231.         </dependency>
  232.         <dependency>
  233.             <groupId>com.fasterxml.jackson.core</groupId>
  234.             <artifactId>jackson-annotations</artifactId>
  235.             <!--<version>2.7.9</version>-->
  236.             <version>2.6.5</version>
  237.         </dependency>
  239.         <!-- https://mvnrepository.com/artifact/com.alibaba/fastjson -->
  240.         <dependency>
  241.             <groupId>com.alibaba</groupId>
  242.             <artifactId>fastjson</artifactId>
  243.             <version>1.2.56</version>
  244.         </dependency>
  246.         <dependency>
  247.            <groupId>org.apache.parquet</groupId>
  248.            <artifactId>parquet-avro</artifactId>
  249.            <version>1.11.1</version>
  250.         </dependency>
  251.         <dependency>
  252.             <groupId>org.apache.avro</groupId>
  253.             <artifactId>avro</artifactId>
  254.             <version>1.10.0</version>
  255.         </dependency>
  256.         <dependency>
  257.            <groupId>org.apache.parquet</groupId>
  258.            <artifactId>parquet-column</artifactId>
  259.            <version>1.11.1</version>
  260.         </dependency>
  262.         <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-hive -->
  263.         <dependency>
  264.             <groupId>org.apache.spark</groupId>
  265.             <artifactId>spark-hive_2.11</artifactId>
  266.             <version>2.4.0</version>
  267.             <scope>provided</scope>
  268.         </dependency>
  270.   </dependencies>
  271. </project>
Javaiceberg

1条答案

按热度按时间
bfrts1fy

bfrts1fy1#

缺少org.apache.parquet.hadoop.columnchunkpagewritestore（org.apache.parquet.hadoop.codecfactory$bytescompressor，org.apache.parquet.schema.messagetype，org.apache.parquet.bytes.bytebufferallocator，int）[java.lang.nosuchmethodexception:org.apache.parquet.hadoop.columnchunkpagewritestore.（org.apache.parquet.hadoop.codecfactory$bytescompressor，org.apache.parquet.schema.messagetype，org.apache.parquet.bytes.bytebufferallocator，int）]
意味着您正在使用columnchunkpagewritestore的构造函数，它接受4个类型的参数（org.apache.parquet.hadoop.codefactory$bytescompressor、org.apache.parquet.schema.messagetype、org.apache.parquet.bytes.bytebufferallocator、int）
它找不到您正在使用的构造函数。这就是为什么没有这样的错误
根据https://jar-download.com/artifacts/org.apache.parquet/parquet-hadoop/1.8.1/source-code/org/apache/parquet/hadoop/columnchunkpagewritestore.java ，你需要1.8.1的Parquethadoop
将mvn导入更改为旧版本。我查看了1.8.1源代码，它有您需要的正确构造函数。

赞(0）分享  回复(0）举报  2021-07-11
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com