我需要从URL下载所有的zip文件:https://www.ercot.com
c0vxltue1#
您所需要的一切都来自一个端点,您可以查询,然后下载所有的zip文件。具体操作如下:
import os import time from pathlib import Path from shutil import copyfileobj import requests endpoint = f"https://www.ercot.com/misapp/servlets/" \ f"IceDocListJsonWS?reportTypeId=11203&_={int(time.time())}" headers = { "Accept": "application/json, text/javascript, */*; q=0.01", "Accept-Encoding": "gzip, deflate, utf-8", "Host": "www.ercot.com", "Referer": "https://www.ercot.com/mp/data-products/data-product-details?id=NP7-802-M", "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:108.0) Gecko/20100101 Firefox/108.0", "X-Requested-With": "XMLHttpRequest" } os.makedirs("zip_files", exist_ok=True) download_url = "https://www.ercot.com/misdownload/servlets/mirDownload?doclookupId=" with requests.Session() as s: auction_results = s.get(endpoint, headers=headers).json() for result in auction_results["ListDocsByRptTypeRes"]["DocumentList"]: file_name = result["Document"]["ConstructedName"] zip_url = f"{download_url}{result['Document']['ReportTypeID']}" print(f"Downloading {result['Document']['FriendlyName']}...") r = s.get(zip_url, headers=headers, stream=True) with open(Path("zip_files") / file_name, 'wb') as f: copyfileobj(r.raw, f)
下载输出:
Downloading 20232nd6AnnualAuctionSeq2CRRAuctionResults... Downloading 20231st6AnnualAuctionSeq1CRRAuctionResults... Downloading 20251st6AnnualAuctionSeq6CRRAuctionResults... ...
这将为您提供一个名为zip_files的文件夹,其中包含:
zip_files
zip_files/ ├── rpt.00011203.0000000000000000.20200103.100105725.20211st6AnnualAuctionSeq3CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20200206.100132142.20212nd6AnnualAuctionSeq4CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20200305.100126722.20221st6AnnualAuctionSeq5CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20200402.100100899.20222nd6AnnualAuctionSeq6CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20200430.100110058.20202nd6AnnualAuctionSeq1CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20200604.100110480.20211st6AnnualAuctionSeq2CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20200702.100115453.20212nd6AnnualAuctionSeq3CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20200806.100118989.20221st6AnnualAuctionSeq4CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20200903.100055873.20222nd6AnnualAuctionSeq5CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20201001.100059887.20231st6AnnualAuctionSeq6CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20201105.100138522.20211st6AnnualAuctionSeq1CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20201203.100105088.20212nd6AnnualAuctionSeq2CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20210105.100208000.20221st6AnnualAuctionSeq3CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20210204.100109061.20222nd6AnnualAuctionSeq4CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20210304.100107618.20231st6AnnualAuctionSeq5CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20210401.100114752.20232nd6AnnualAuctionSeq6CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20210506.100121795.20212nd6AnnualAuctionSeq1CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20210603.100129464.20221st6AnnualAuctionSeq2CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20210701.100127848.20222nd6AnnualAuctionSeq3CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20210805.100115654.20231st6AnnualAuctionSeq4CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20210902.100111410.20232nd6AnnualAuctionSeq5CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20210930.100118983.20241st6AnnualAuctionSeq6CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20211104.100126210.20221st6AnnualAuctionSeq1CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20211202.100129217.20222nd6AnnualAuctionSeq2CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20220104.100123693.20231st6AnnualAuctionSeq3CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20220203.100118822.20232nd6AnnualAuctionSeq4CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20220303.100110243.20241st6AnnualAuctionSeq5CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20220331.100104447.20242nd6AnnualAuctionSeq6CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20220505.100109570.20222nd6AnnualAuctionSeq1CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20220602.100104646.20231st6AnnualAuctionSeq2CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20220630.100107920.20232nd6AnnualAuctionSeq3CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20220804.100105512.20241st6AnnualAuctionSeq4CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20220901.100103704.20242nd6AnnualAuctionSeq5CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20221006.100123996.20251st6AnnualAuctionSeq6CRRAuctionResults.zip ├── rpt.00011203.0000000000000000.20221103.100114498.20231st6AnnualAuctionSeq1CRRAuctionResults.zip └── rpt.00011203.0000000000000000.20221201.130125954.20232nd6AnnualAuctionSeq2CRRAuctionResults.zip
1条答案
按热度按时间c0vxltue1#
您所需要的一切都来自一个端点,您可以查询,然后下载所有的zip文件。
具体操作如下:
下载输出:
这将为您提供一个名为
zip_files
的文件夹,其中包含: