from datetime import datetime, timedelta
start_date=datetime.strptime("2021-01-01", "%Y-%m-%d")
days = 30
paths=[]
for i in range(0, days+1):
oper_date=start_date - timedelta(days=i)
path = "table/{year}/{month}/{day}".format(year=oper_date.year, month=oper_date.month, day = oper_date.day)
paths.append(path)
spark.read.parquet(*paths)
1条答案
按热度按时间7fhtutme1#
由于可以传递目标路径列表,因此首先可以在读取文件之前生成路径。下面是一个示例代码,假设输入数据为Parquet格式: