我试图从一个Dataframe中读取数据并在另一个Dataframe中使用它。我怎么才能优雅地做呢?
val query = s"select distinct p_id, lower(regexp_replace(p_id,'[^a-zA-Z0-9]+','_')) as p_id_formatted, lower(regexp_extract(f_id,'^([^\\.]+)\\.?',1)) as f_id_formatted, column_name from default.rc_pcoders"
val run_query = sql(query)
val table_name = run_query.select(concat(lit("nepp"), lit("_"),$"p_id_formatted", lit("_") ,$"f_id_formatted ").alias("tablename"),$"column_name")
这给了我下面的输出,它本质上代表了一个表名
+------------------+-----------+
|tablename |column_name|
+------------------+-----------+
|nepp_148hl16011_cm|cmtrt |
|nepp_148hl16011_mh|mhaspe |
|nepp_148hl16011_ae|aeputt |
+------------------+-----------+
如何从每个表中获取列名?类似于(下面的查询不起作用)
val whole_query = sql("show columns in "table_name.tablename"")
1条答案
按热度按时间nhn9ugyo1#
首先,收集要加载的表的所有名称:
其次,获取对相应Dataframe的引用,将它们与它们的列名相关联