我有以下Dataframe:
dataframe1
+-----------------------+
|ID |
+-----------------------+
|[10,80,60,] |
|[20,40,] |
+-----------------------+
另一个Dataframe:
dataframe2
+------------------+----------------+
|ID_2 | name |
+------------------+----------------+
|40 | XYZZ |
|200 | vbb |
+------------------+----------------+
我想要以下输出:
+------------------+----------------+
|ID_2 | name |
+------------------+----------------+
|40 | XYZZ |
+------------------+----------------+
我使用下面的代码从第二个 dataframe
行witch id_2==id。
for (java.util.Iterator<Row> iter = dataframe1.toLocalIterator(); iter.hasNext();) {
String item = (iter.next()).get(0).toString();
dataframe2.registerTempTable("data2");
Dataset<Row> res = sparkSession.sql("select * from data2 where ID_2 IN ("+item+")");
res.show();
}
但我有以下例外:
Exception in thread "main" org.apache.spark.sql.catalyst.parser.ParseException:
mismatched input 'from' expecting <EOF>(line 1, pos 9)
== SQL ==
select * from data2 where ID_2 IN ([10,80,60,])
---------^^^
at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:241)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:117)
at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:48)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:69)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
at factory.Geofencing_Alert.check(Geofencing_Alert.java:84)
at factory.Geofencing_Alert.main(Geofencing_Alert.java:158)
我怎样才能解决这个问题?
1条答案
按热度按时间gab6jxml1#
只需使用
explode
功能。