druid hadoop批处理主管:无法将类型id'index.hadoop'解析为supervisorspec的子管道

vu8f3i0k  于 2021-05-27  发布在  Hadoop
关注(0)|答案(1)|浏览(348)

我正在尝试启动一个druid管理器来接收存储在hadoop中的parqurt数据。然而,我得到以下错误,我找不到任何有关它的信息:
“error”:“无法将类型id'index\u hadoop'解析为[simple type,class io.druid.indexing.overlord.supervisor.supervisorspec]:已知类型id=[noopsuperservisorspec,kafka]\n位于[source:(org.eclipse.jetty.server.httpinputoverhttp)
我尝试在扩展加载列表中加载hadoop deep storage、parquet和avro扩展来修复它,但是没有成功。
这是我的主管json配置:

{
  "type" : "index_hadoop",
  "spec" : {
    "dataSchema" : {
      "dataSource" : "hadoop-batch-timeseries",
      "parser" : {
        "type": "parquet",
        "parseSpec" : {
            "format" : "parquet",
            "flattenSpec": {
                "useFieldDiscovery": true,
                "fields": [
                ]
            },
            "timestampSpec" : {
                "column" : "timestamp",
                "format" : "auto"
            },
            "dimensionsSpec" : {
                "dimensions": [ "installation", "var_id", "value" ],
                "dimensionExclusions" : [],
                "spatialDimensions" : []
            }
        }
      },
      "metricsSpec" : [
        {
          "type" : "count",
          "name" : "count"
        }
      ],
      "granularitySpec" : {
        "type" : "uniform",
        "segmentGranularity" : "DAY",
        "queryGranularity" : "NONE",
        "intervals" : [ "2018-10-01/2018-11-30" ]
      }
    },
    "ioConfig": {
      "type": "hadoop",
      "inputSpec": {
        "type": "granularity",
        "dataGranularity": "day",
        "inputFormat": "org.apache.druid.data.input.parquet.DruidParquetInputFormat",
        "inputPath": "/warehouse/tablespace/external/hive/demo.db/integers",
        "filePattern": "*.parquet",
        "pathFormat": "'year'=yyy/'month'=MM/'day'=dd"
      },
    },
    "tuningConfig" : {
      "type": "hadoop"
    }
  },
  "hadoopDependencyCoordinates": "3.1.0"
}
disho6za

disho6za1#

我也遇到了同样的问题。通过将其作为任务提交而不是作为主管提交来解决:

curl -X POST -H 'Content-Type: application/json' -d @my-spec.json http://my-druid-coordinator-url:8081/druid/indexer/v1/task

相关问题