添加基于符合性分析器的检查

8tntrjer  于 2021-05-27  发布在  Spark
关注(0)|答案(2)|浏览(364)

下面是我正在使用的示例Dataframe(df):

+---+----+--------+
| id|orig|scrubbed|
+---+----+--------+
|  1|   a|       a|
|  2|   B|       b|
|  3|   c|       c|
|  4|   D|       d|
|  5|   *|      XX|
|  6|   $|      XX|
|  7|  ZZ|      ZZ|
|  8|  XX|      XX|
|  9|   y|       y|
| 10|   Z|       z|
+---+----+--------+

我想执行一个检查,告诉我在擦洗之后“填充”的项目(不包含“xx”或“zz”)的比例是否至少为80%(此检查应失败。)我可以将符合性分析器添加到 VerificationRunBuilder 计算指标,如下所示:

val myVerificationResult: VerificationResult = new VerificationRunBuilder(df).
    addRequiredAnalyzer(
        Compliance(
            "populatedAfterScrubbing",
            "`scrubbed` NOT IN ('ZZ', 'XX') AND `scrubbed` IS NOT NULL",
            Some("`orig` NOT IN ('ZZ', 'XX') AND `orig` IS NOT NULL")
        )
    ).
    addCheck(
        Check(CheckLevel.Error, "Review Check").
            hasSize(_ >= 1)
    ).
    run()

此代码运行并使用 hasSize 约束,但我不知道如何添加基于自定义符合性分析器的约束。这可能吗?

qnzebej0

qnzebej01#

我找到了一个似乎有效的解决方案,以防有人感兴趣。答案在于创建自定义约束而不是自定义分析器。以下是工作代码:

val myConstraint = Constraint.complianceConstraint(
    "my constraint",
    "`scrubbed` NOT IN ('ZZ', 'XX') AND `scrubbed` IS NOT NULL",
    (fraction:Double)=>fraction>=0.8,
    Some("`orig` NOT IN ('ZZ', 'XX') AND `orig` IS NOT NULL"),
    Some("no peeking")
)

val myVerificationResult: VerificationResult = { VerificationSuite()
    .onData(df)
    .addCheck(
        Check(CheckLevel.Error, "Review Check") 
            .addConstraint(myConstraint)
    )
    .run()
}
val result = checkResultsAsDataFrame(spark, myVerificationResult)
result.show(truncate=true)

结果与预期完全一致:

+------------+-----------+------------+--------------------+-----------------+--------------------+
|       check|check_level|check_status|          constraint|constraint_status|  constraint_message|
+------------+-----------+------------+--------------------+-----------------+--------------------+
|Review Check|      Error|       Error|ComplianceConstra...|          Failure|Value: 0.75 does ...|
+------------+-----------+------------+--------------------+-----------------+--------------------+
vltsax25

vltsax252#

这不能只通过使用统计进行检查吗https://github.com/awslabs/deequ/blob/master/src/main/scala/com/amazon/deequ/checks/check.scala#l667 像这样的

Check(CheckLevel.Warning, "Statisfies TEST Constraint")      
      .satisfies("`scrubbed` NOT IN ('ZZ', 'XX') AND `scrubbed` IS NOT NULL",
                 "my constraint",
                 "fraction:Double",(fraction:Double)=>fraction>=0.8,
                  Some("..."))
      ))

我认为这是oob,而不是通过合规约束来定义,虽然这也是一个食品的想法,如果你有一个复杂的逻辑。

相关问题