在datalab中查询配置单元表的问题

jv2fixgn 于 2021-06-24 发布在 Hive

关注(0)|答案(2)|浏览(272)

我已经创建了一个dataproc集群，使用更新的init操作来安装datalab。
一切正常，只是当我从datalab笔记本查询配置单元表时，遇到了

hc.sql(“””select * from invoices limit 10”””)

"java.lang.ClassNotFoundException: Class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem not found" exception

创建群集

gcloud beta dataproc clusters create ds-cluster \
--project my-exercise-project \
--region us-west1 \
--zone us-west1-b \
--bucket dataproc-datalab \
--scopes cloud-platform  \
--num-workers 2  \
--enable-component-gateway  \
--initialization-actions gs://dataproc_mybucket/datalab-updated.sh,gs://dataproc-initialization-actions/connectors/connectors.sh  \
--metadata 'CONDA_PACKAGES="python==3.5"'  \
--metadata gcs-connector-version=1.9.11

数据实验室更新.sh

-v "${DATALAB_DIR}:/content/datalab" ${VOLUME_FLAGS} datalab-pyspark; then
    mkdir -p ${HOME}/datalab
    gcloud source repos clone datalab-notebooks ${HOME}/datalab/notebooks

在数据实验室笔记本里

from pyspark.sql import HiveContext
hc=HiveContext(sc)
hc.sql("""show tables in default""").show()
hc.sql(“””CREATE EXTERNAL TABLE IF NOT EXISTS INVOICES
      (SubmissionDate DATE, TransactionAmount DOUBLE, TransactionType STRING)
      STORED AS PARQUET
      LOCATION 'gs://my-exercise-project-ds-team/datasets/invoices’”””)
hc.sql(“””select * from invoices limit 10”””)

更新

spark._jsc.hadoopConfiguration().set('fs.gs.impl', 'com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem')
spark._jsc.hadoopConfiguration().set('fs.gs.auth.service.account.enable', 'true')
spark._jsc.hadoopConfiguration().set('google.cloud.auth.service.account.json.keyfile', "~/Downloads/my-exercise-project-f47054fc6fd8.json")

更新2（datalab-updated.sh）

function run_datalab(){
  if docker run -d --restart always --net=host  \
      -v "${DATALAB_DIR}:/content/datalab" ${VOLUME_FLAGS} datalab-pyspark; then
    mkdir -p ${HOME}/datalab
    gcloud source repos clone datalab-notebooks ${HOME}/datalab/notebooks
    echo 'Cloud Datalab Jupyter server successfully deployed.'
  else
    err 'Failed to run Cloud Datalab'
  fi
}

Hive google-cloud-dataproc google-cloud-datalab

来源：https://stackoverflow.com/questions/55944773/issue-querying-a-hive-table-in-datalab

2条答案

按热度按时间

oalqel3c1#

如果要在datalab中使用配置单元，则必须启用配置单元元存储

--properties hive:hive.metastore.warehouse.dir=gs://$PROJECT-warehouse/datasets \
--metadata "hive-metastore-instance=$PROJECT:$REGION:hive-metastore"

在你的情况下

--properties hive:hive.metastore.warehouse.dir=gs://$PROJECT-warehouse/datasets \
--metadata "hive-metastore-instance=$PROJECT:$REGION:hive-metastore"

hc.sql(“””CREATE EXTERNAL TABLE IF NOT EXISTS INVOICES
      (SubmissionDate DATE, TransactionAmount DOUBLE, TransactionType STRING)
      STORED AS PARQUET
      LOCATION 'gs://$PROJECT-warehouse/datasets/invoices’”””)

并确保添加以下设置以启用gcs

sc._jsc.hadoopConfiguration().set('fs.gs.impl', 'com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem')

# This is required if you are using service account and set true,

sc._jsc.hadoopConfiguration().set('fs.gs.auth.service.account.enable', 'false')
sc._jsc.hadoopConfiguration().set('google.cloud.auth.service.account.json.keyfile', "/path/to/keyfile")

# Following are required if you are using oAuth

sc._jsc.hadoopConfiguration().set('fs.gs.auth.client.id', 'YOUR_OAUTH_CLIENT_ID')
sc._jsc.hadoopConfiguration().set('fs.gs.auth.client.secret', 'OAUTH_SECRET')

赞(0）回复(0）举报 2021-06-24

kupeojn62#

应使用datalab初始化操作在dataproc群集上安装datalab：

gcloud dataproc clusters create ${CLUSTER} \
    --image-version=1.3 \
    --scopes cloud-platform \
    --initialization-actions=gs://dataproc-initialization-actions/datalab/datalab.sh

此配置单元与数据实验室中的gcs一起工作后：

from pyspark.sql import HiveContext
hc=HiveContext(sc)
hc.sql("""SHOW TABLES IN default""").show()

输出：

+--------+---------+-----------+
|database|tableName|isTemporary|
+--------+---------+-----------+
+--------+---------+-----------+

使用datalab中的配置单元在gcs上创建外部表：

hc.sql("""CREATE EXTERNAL TABLE IF NOT EXISTS INVOICES
      (SubmissionDate DATE, TransactionAmount DOUBLE, TransactionType STRING)
      STORED AS PARQUET
      LOCATION 'gs://<BUCKET>/datasets/invoices'""")

输出：

DataFrame[]

使用datalab中的配置单元查询gcs表：

hc.sql("""SELECT * FROM invoices LIMIT 10""")

输出：

DataFrame[SubmissionDate: date, TransactionAmount: double, TransactionType: string]

赞(0）回复(0）举报 2021-06-24

我来回答

在datalab中查询配置单元表的问题

创建群集

数据实验室更新.sh

在数据实验室笔记本里

更新2（datalab-updated.sh）

2条答案

相关问题

热门标签

最新问答