我在pyspark中实现了一个文本分类器,如下所示
tokenizer = RegexTokenizer(inputCol="documents", outputCol="tokens", pattern='\\W+')
remover = StopWordsRemover(inputCol='tokens', outputCol='nostops')
vectorizer = CountVectorizer(inputCol='nostops', outputCol='features', vocabSize=1000)
labelIndexer = StringIndexer(inputCol="label", outputCol="indexedLabel", handleInvalid='skip')
labelIndexer_model = labelIndexer.fit(countModel_df)
convertor = IndexToString(inputCol='prediction', outputCol='predictedLabel', labels=labelIndexer_model.labels)
rfc = RandomForestClassifier(featuresCol='features', labelCol='indexedLabel', numTrees=30)
evaluator = BinaryClassificationEvaluator(labelCol='indexedLabel', rawPredictionCol='prediction')
pipe_rfc = Pipeline(stages=[tokenizer, remover, labelIndexer, vectorizer, rfc, convertor])
train_df, test_df = df.randomSplit((0.8, 0.2), seed=42)
model = pipe_rfc.fit(train_df)
prediction_rfc_df = rfc_model.transform(test_df)
代码正在工作,预测函数按预期进行预测。但是当我想检查元数据时,元数据字典是空的,如下所示
prediction_rfc_df.schema['features'].metadata
Output : {}
prediction_rfc_df.schema['label'].metadata
Output: {}
你知道为什么数据框中缺少元数据吗?
我从Cassandra表格中读取数据如下:
df = spark.read \
.format("org.apache.spark.sql.cassandra") \
.options(table='table_name', keyspace='key_space_name') \
.load()
暂无答案!
目前还没有任何答案,快来回答吧!