scala—将行值从spark df传递到函数

4urapxun 于 2021-05-19 发布在 Spark

关注(0)|答案(1)|浏览(504)

val someDF = Seq(
  (8, "abc"),
  (64, "sdf"),
  (-27, "yui")
).toDF("number", "word")

def numberValidation(numberValidation:Any): Boolean=
numberValidation match{
case int :Integer => true
case _ => false}

def wordValidatoin(wordValidation:Any): Boolean=
wordValidation match{
case str: String => true
case _ => false}

我需要将每一行及其coresponding值传递给上述函数。第一行数字列的值=8，传递给numbervalidation，第一行单词列的值传递给wordvalidatoin

scala apache-spark apache-spark-sql

来源：https://stackoverflow.com/questions/64545140/passing-row-values-from-spark-df-to-functions

1条答案

按热度按时间

wfveoks01#

使用spark自定义项并按以下方式更改方法：

val numberValidation = udf((numberValidation: Any) => {
     numberValidation match{
    case int :Integer => true
    case _ => false
    }
  })

val wordValidatoin = udf((wordValidation:Any)): =>{
    wordValidation match{
    case str: String => true
    case _ => false
    }
})

并与df核对如下：

someDF
.withColumn("checkNum",numberValidation($"number"))
.withColumn("checWord",wordValidatoin($"word"))
.show

赞(0）回复(0）举报 2021-05-19

我来回答

scala—将行值从spark df传递到函数

1条答案

相关问题

热门标签

最新问答