线程“main”org.apache.spark.sql.catalyst.parser.parseexception中的异常

fzwojiic  于 2021-05-27  发布在  Spark
关注(0)|答案(1)|浏览(978)

我有以下Dataframe:

dataframe1
+-----------------------+
|ID                     |
+-----------------------+
|[10,80,60,]            |
|[20,40,]               |
+-----------------------+

另一个Dataframe:

dataframe2
+------------------+----------------+
|ID_2              |   name         |
+------------------+----------------+
|40                | XYZZ           |
|200               | vbb            |
+------------------+----------------+

我想要以下输出:

+------------------+----------------+
|ID_2              |   name         |
+------------------+----------------+
|40                | XYZZ           |
+------------------+----------------+

我使用下面的代码从第二个 dataframe 行witch id_2==id。

for (java.util.Iterator<Row> iter = dataframe1.toLocalIterator(); iter.hasNext();) {
        String item = (iter.next()).get(0).toString();
        dataframe2.registerTempTable("data2");
        Dataset<Row> res = sparkSession.sql("select * from data2 where ID_2 IN ("+item+")");
        res.show();
}

但我有以下例外:

Exception in thread "main" org.apache.spark.sql.catalyst.parser.ParseException: 
mismatched input 'from' expecting <EOF>(line 1, pos 9)

 == SQL ==
select * from data2 where ID_2 IN ([10,80,60,])
 ---------^^^

at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:241)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:117)
at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:48)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:69)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
at factory.Geofencing_Alert.check(Geofencing_Alert.java:84)
at factory.Geofencing_Alert.main(Geofencing_Alert.java:158)

我怎样才能解决这个问题?

gab6jxml

gab6jxml1#

只需使用 explode 功能。

df1.withColumn("ID", explode($"ID"))
  .join(df2, $"ID" === $"ID_2", "inner")
  .drop("ID")
  .show

+----+----+
|ID_2|name|
+----+----+
|  40|xyzz|
+----+----+

相关问题