presto在localhost上寻找分区，而不是在具有hive metastore的远程主机上

balp4ylt  于 2021-06-26  发布在  Hive
关注(0)|答案(0)|浏览(258)

hosta安装并运行了mysql（3306端口）、hive（10000端口）和hivemetastore（9083端口）。hostb已安装并运行presto。
目标是让hostb运行presto，它允许对hosta上的hivemetastore进行查询。
下面是获取错误/home/ec2 user/warehouse/contact在hosta的本地文件系统（不是hdfs/s3）上确实存在（并且表已分区），但在hostb上却不存在，为什么presto要在运行presto的localhost（hostb）上而不是在hosta（hive metastore所在的位置）上查找配置单元分区呢？元存储连接是建立的，因为presto能够列出元存储上的表，。

  1. presto-cli --debug --catalog hive --schema default
  2. presto:default> show tables;
  3.            Table
  4. ----------------------------
  5.  account
  6.  contact
  7. (2 rows)
  9. Query 20171102_122934_00012_x6ppj, FINISHED, 2 nodes
  10. http://localhost:8080/query.html?20171102_122934_00012_x6ppj
  11. Splits: 18 total, 18 done (100.00%)
  12. CPU Time: 0.0s total,   615 rows/s, 18.8KB/s, 5% active
  13. Per Node: 0.0 parallelism,     8 rows/s,   280B/s
  14. Parallelism: 0.0
  15. 0:00 [8 rows, 250B] [17 rows/s, 560B/s]
  17. presto:default> select * from contact;
  18. Query 20171102_122943_00013_x6ppj failed: Partition location does not exist: file:/home/ec2-user/warehouse/contact
  19. com.facebook.presto.spi.PrestoException: Partition location does not exist: file:/home/ec2-user/warehouse/contact
  20.         at com.facebook.presto.hive.util.HiveFileIterator.computeNext(HiveFileIterator.java:102)
  21.         at com.facebook.presto.hive.util.HiveFileIterator.computeNext(HiveFileIterator.java:41)
  22.         at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)
  23.         at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140)
  24.         at com.facebook.presto.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:243)
  25.         at com.facebook.presto.hive.BackgroundHiveSplitLoader.access$300(BackgroundHiveSplitLoader.java:92)
  26.         at com.facebook.presto.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:195)
  27.         at com.facebook.presto.hive.util.ResumableTasks.safeProcessTask(ResumableTasks.java:45)
  28.         at com.facebook.presto.hive.util.ResumableTasks.lambda$submit$1(ResumableTasks.java:33)
  29.         at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)
  30.         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  31.         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  32.         at java.lang.Thread.run(Thread.java:748)
  34. cat config.properties
  35. coordinator=true
  36. node-scheduler.include-coordinator=false
  37. http-server.http.port=8080
  38. query.max-memory=50GB
  39. query.max-memory-per-node=1GB
  40. discovery-server.enabled=true
  42. # discovery.uri=http://example.net:8080
  44. discovery.uri=http://hostB:8080
  46. cat hive.properties
  47. connector.name=hive-hadoop2
  48. hive.metastore.uri=thrift://hostA:9083
  50. 2017-11-02T06:52:30.585Z        INFO    main    com.facebook.presto.metadata.StaticCatalogStore -- Loading catalog etc/catalog/hive.properties --
  51. 2017-11-02T06:52:31.307Z        INFO    main    Bootstrap       PROPERTY                                           DEFAULT     RUNTIME                        DESCRIPTION
  52. 2017-11-02T06:52:31.307Z        INFO    main    Bootstrap       hive.allow-corrupt-writes-for-testing              false       false                          Allow Hive connector to write data even when data will likely be corrupt
  53. 2017-11-02T06:52:31.307Z        INFO    main    Bootstrap       hive.assume-canonical-partition-keys               false       false
  54. 2017-11-02T06:52:31.307Z        INFO    main    Bootstrap       hive.bucket-execution                              true        true                           Enable bucket-aware execution: only use a single worker per bucket
  55. 2017-11-02T06:52:31.307Z        INFO    main    Bootstrap       hive.bucket-writing                                true        true                           Enable writing to bucketed tables
  56. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.dfs.connect.max-retries                       5           5
  57. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.dfs.connect.timeout                           500.00ms    500.00ms
  58. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.dfs-timeout                                   60.00s      60.00s
  59. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.domain-compaction-threshold                   100         100                            Maximum ranges to allow in a tuple domain without compacting it
  60. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.dfs.domain-socket-path                        null        null
  61. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.fs.cache.max-size                             1000        1000                           Hadoop FileSystem cache size
  62. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.force-local-scheduling                        false       false
  63. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.hdfs.authentication.type                      NONE        NONE                           HDFS authentication type
  64. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.hdfs.impersonation.enabled                    false       false                          Should Presto user be impersonated when communicating with HDFS
  65. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.compression-codec                             GZIP        GZIP
  66. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.metastore.authentication.type                 NONE        NONE                           Hive Metastore authentication type
  67. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.storage-format                                RCBINARY    RCBINARY
  68. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.immutable-partitions                          false       false                          Can new data be inserted into existing partitions or existing unpartitioned tables
  69. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.dfs.ipc-ping-interval                         10.00s      10.00s
  70. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.max-concurrent-file-renames                   20          20
  71. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.max-initial-split-size                        32MB        32MB
  72. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.max-initial-splits                            200         200
  73. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.metastore-refresh-max-threads                 100         100
  74. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.max-outstanding-splits                        1000        1000
  75. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.metastore.partition-batch-size.max            100         100
  76. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.max-partitions-per-scan                       100000      100000                         Maximum allowed partitions for a single table scan
  77. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.max-partitions-per-writers                    100         100                            Maximum number of partitions per writer
  78. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.max-split-iterator-threads                    1000        1000
  79. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.max-split-size                                64MB        64MB
  80. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.metastore-cache-maximum-size                  10000       10000
  81. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.metastore-cache-ttl                           0.00s       0.00s
  82. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.metastore-refresh-interval                    0.00s       0.00s
  83. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.metastore.thrift.client.socks-proxy           null        null
  84. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.metastore-timeout                             10.00s      10.00s
  85. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.metastore.partition-batch-size.min            10          10
  86. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.orc.bloom-filters.enabled                     false       false
  87. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.orc.default-bloom-filter-fpp                  0.05        0.05                           ORC Bloom filter false positive probability
  88. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.orc.max-buffer-size                           8MB         8MB
  89. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.orc.max-merge-distance                        1MB         1MB
  90. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.orc.max-read-block-size                       16MB        16MB
  91. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.orc.optimized-writer.enabled                  false       false
  92. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.orc.stream-buffer-size                        8MB         8MB
  93. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.parquet-optimized-reader.enabled              false       false
  94. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.parquet-predicate-pushdown.enabled            false       false
  95. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.per-transaction-metastore-cache-maximum-size  1000        1000
  96. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.rcfile-optimized-writer.enabled               true        true
  97. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.rcfile.writer.validate                        false       false                          Validate RCFile after write by re-reading the whole file
  98. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.recursive-directories                         false       false
  99. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.config.resources                              null        null
  100. 2017-11-02T06:52:31.309Z        INFO    main    Bootstrap       hive.respect-table-format                          true        true                           Should new partitions be written using the existing table format or the default Presto format
  101. 2017-11-02T06:52:31.310Z        INFO    main    Bootstrap       hive.skip-deletion-for-alter                       false       false                          Skip deletion of old partition data when a partition is deleted and then inserted in the same transaction
  102. 2017-11-02T06:52:31.310Z        INFO    main    Bootstrap       hive.table-statistics-enabled                      true        true                           Enable use of table statistics
  103. 2017-11-02T06:52:31.310Z        INFO    main    Bootstrap       hive.time-zone                                     Zulu        Zulu
  104. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.orc.use-column-names                          false       false                          Access ORC columns using names from the file
  105. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.parquet.use-column-names                      false       false                          Access Parquet columns using names from the file
  106. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.dfs.verify-checksum                           true        true
  107. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.write-validation-threads                      16          16                             Number of threads used for verifying data after a write
  108. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.non-managed-table-writes-enabled              false       false                          Enable writes to non-managed (external) tables
  109. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.s3.pin-client-to-current-region               false       false                          Should the S3 client be pinned to the current EC2 region
  110. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.s3.aws-access-key                             null        null
  111. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.s3.aws-secret-key                             [REDACTED]  [REDACTED]
  112. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.s3.connect-timeout                            5.00s       5.00s
  113. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.s3.encryption-materials-provider              null        null                           Use a custom encryption materials provider for S3 data encryption
  114. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.s3.endpoint                                   null        null
  115. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.s3.kms-key-id                                 null        null                           Use an AWS KMS key for S3 data encryption
  116. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.s3.max-backoff-time                           10.00m      10.00m
  117. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.s3.max-client-retries                         5           5
  118. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.s3.max-connections                            500         500
  119. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.s3.max-error-retries                          10          10
  120. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.s3.max-retry-time                             10.00m      10.00m
  121. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.s3.multipart.min-file-size                    16MB        16MB                           Minimum file size for an S3 multipart upload
  122. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.s3.multipart.min-part-size                    5MB         5MB                            Minimum part size for an S3 multipart upload
  123. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.s3.signer-type                                null        null
  124. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.s3.socket-timeout                             5.00s       5.00s
  125. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.s3.sse.enabled                                false       false                          Enable S3 server side encryption
  126. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.s3.sse.kms-key-id                             null        null                           KMS Key ID to use for S3 server-side encryption with KMS-managed key
  127. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.s3.sse.type                                   S3          S3                             Key management type for S3 server-side encryption (S3 or KMS)
  128. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.s3.ssl.enabled                                true        true
  129. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.s3.staging-directory                          /tmp        /tmp                           Temporary directory for staging files before uploading to S3
  130. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.s3.use-instance-credentials                   true        true
  131. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.s3.user-agent-prefix                                                                     The user agent prefix to use for S3 calls
  132. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.metastore.uri                                 null        [thrift://hostA:9083]  Hive metastore URIs (comma separated)
  133. 2017-11-02T06:52:31.311Z        INFO    main    Bootstrap       hive.metastore                                     thrift      thrift
  134. 2017-11-02T06:52:31.312Z        INFO    main    Bootstrap       hive.allow-add-column                              false       false                          Allow Hive connector to add column
  135. 2017-11-02T06:52:31.312Z        INFO    main    Bootstrap       hive.allow-drop-column                             false       false                          Allow Hive connector to drop column
  136. 2017-11-02T06:52:31.312Z        INFO    main    Bootstrap       hive.allow-drop-table                              false       false                          Allow Hive connector to drop table
  137. 2017-11-02T06:52:31.312Z        INFO    main    Bootstrap       hive.allow-rename-column                           false       false                          Allow Hive connector to rename column
  138. 2017-11-02T06:52:31.312Z        INFO    main    Bootstrap       hive.allow-rename-table                            false       false                          Allow Hive connector to rename table
  139. 2017-11-02T06:52:31.312Z        INFO    main    Bootstrap       hive.security                                      legacy      legacy
  140. 2017-11-02T06:52:31.312Z        INFO    main    Bootstrap
  141. 2017-11-02T06:52:32.663Z        INFO    main    com.facebook.presto.metadata.StaticCatalogStore -- Added catalog hive using connector hive-hadoop2 --
Hiveprestoapache-rangerapachethrift

暂无答案！

目前还没有任何答案，快来回答吧！

我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备13028337号-1 大数据知识库 https://www.saoniuhuo.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com