所以我有一个代码，它从14页中剔除了矿物的名称和价格（到目前为止）并将其保存到. txt文件。我首先只尝试了Page1，然后我想添加更多的页面以容纳更多的数据。但随后代码抓取了一些不应该抓取的东西--一个随机名称/字符串。我没想到它会抓取那个，但它确实做到了，并且给它指定了一个错误的价格!这种情况发生在一个具有这个"意外名称"的矿物之后，然后整个列表的其余部分都有错误的价格。见下图：
因此，由于此字符串不同于其他任何字符串，因此进一步的代码无法拆分它并给出错误：
cutted2 = split2.pop(1)
^^^^^^^^^^^^^
IndexError: pop index out of range我试图忽略这些错误，并使用不同的Stackoverflow页面中使用的方法之一：
try:
cutted2 = split2.pop(1)
except IndexError:
continue它确实工作了，没有出现错误...但是它给错误的矿物分配了错误的价格（正如我注意到的）!!!我怎么能改变代码，只是忽略这些"奇怪"的名字，只是继续与列表？下面是整个代码，它停止在URL5，因为我记得，并给出了这个弹出索引错误：
import requests
from bs4 import BeautifulSoup
import re
def collecter(URL):
headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36"}
soup = BeautifulSoup(requests.get(URL, headers=headers).text, "lxml")
names = [n.getText(strip=True) for n in soup.select("table tr td font a")]
prices = [
p.getText(strip=True).split("Price:")[-1] for p
in soup.select("table tr td font font")
]
names[:] = [" ".join(n.split()) for n in names if not n.startswith("[")]
prices[:] = [p for p in prices if p]
with open("Minerals.txt", "a+", encoding='utf-8') as file:
for name, price in zip(names, prices):
# print(f"{name}\n{price}")
# print("-" * 50)
filename = str(name)+" "+str(price)+"\n"
split1 = filename.split(' / ')
cutted1 = split1.pop(0)
split2 = cutted1.split(": ")
try:
cutted2 = split2.pop(1)
except IndexError:
continue
two_prices = cutted2+" "+split1.pop(0)+"\n"
file.write(two_prices)
URL1 = "https://www.fabreminerals.com/search_results.php?LANG=EN&SearchTerms=&submit=Buscar&MineralSpeciment=&Country=&Locality=&PriceRange=&checkbox=enventa&First=0"
URL2 = "https://www.fabreminerals.com/search_results.php?LANG=EN&SearchTerms=&submit=Buscar&MineralSpeciment=&Country=&Locality=&PriceRange=&checkbox=enventa&First=25"
URL3 = "https://www.fabreminerals.com/search_results.php?LANG=EN&SearchTerms=&submit=Buscar&MineralSpeciment=&Country=&Locality=&PriceRange=&checkbox=enventa&First=50"
URL4 = "https://www.fabreminerals.com/search_results.php?LANG=EN&SearchTerms=&submit=Buscar&MineralSpeciment=&Country=&Locality=&PriceRange=&checkbox=enventa&First=75"
URL5 = "https://www.fabreminerals.com/search_results.php?LANG=EN&SearchTerms=&submit=Buscar&MineralSpeciment=&Country=&Locality=&PriceRange=&checkbox=enventa&First=100"
URL6 = "https://www.fabreminerals.com/search_results.php?LANG=EN&SearchTerms=&submit=Buscar&MineralSpeciment=&Country=&Locality=&PriceRange=&checkbox=enventa&First=125"
URL7 = "https://www.fabreminerals.com/search_results.php?LANG=EN&SearchTerms=&submit=Buscar&MineralSpeciment=&Country=&Locality=&PriceRange=&checkbox=enventa&First=150"
URL8 = "https://www.fabreminerals.com/search_results.php?LANG=EN&SearchTerms=&submit=Buscar&MineralSpeciment=&Country=&Locality=&PriceRange=&checkbox=enventa&First=175"
URL9 = "https://www.fabreminerals.com/search_results.php?LANG=EN&SearchTerms=&submit=Buscar&MineralSpeciment=&Country=&Locality=&PriceRange=&checkbox=enventa&First=200"
URL10 = "https://www.fabreminerals.com/search_results.php?LANG=EN&SearchTerms=&submit=Buscar&MineralSpeciment=&Country=&Locality=&PriceRange=&checkbox=enventa&First=225"
URL11 = "https://www.fabreminerals.com/search_results.php?LANG=EN&SearchTerms=&submit=Buscar&MineralSpeciment=&Country=&Locality=&PriceRange=&checkbox=enventa&First=250"
URL12 = "https://www.fabreminerals.com/search_results.php?LANG=EN&SearchTerms=&submit=Buscar&MineralSpeciment=&Country=&Locality=&PriceRange=&checkbox=enventa&First=275"
URL13 = "https://www.fabreminerals.com/search_results.php?LANG=EN&SearchTerms=&submit=Buscar&MineralSpeciment=&Country=&Locality=&PriceRange=&checkbox=enventa&First=300"
URL14 = "https://www.fabreminerals.com/search_results.php?LANG=EN&SearchTerms=&submit=Buscar&MineralSpeciment=&Country=&Locality=&PriceRange=&checkbox=enventa&First=325"
collecter(URL1)
collecter(URL2)
collecter(URL3)
collecter(URL4)
collecter(URL5)
collecter(URL6)
collecter(URL7)
collecter(URL8)
collecter(URL9)
collecter(URL10)
collecter(URL11)
collecter(URL12)
collecter(URL13)
collecter(URL14)编辑：这是完整的工作代码下面，谢谢帮助的家伙!
import requests
from bs4 import BeautifulSoup
import re
for URL in range(0,2569,25):
headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36"}
soup = BeautifulSoup(requests.get(f'https://www.fabreminerals.com/search_results.php?LANG=EN&SearchTerms=&submit=Buscar&MineralSpeciment=&Country=&Locality=&PriceRange=&checkbox=enventa&First={URL}', headers=headers).text, "lxml")
names = [n.getText(strip=True) for n in soup.select("table tr td font>a")]
prices = [p.getText(strip=True).split("Price:")[-1] for p in soup.select("table tr td font>font")]
names[:] = [" ".join(n.split()) for n in names if not n.startswith("[") ]
prices[:] = [p for p in prices if p]
with open("MineralsList.txt", "a+", encoding='utf-8') as file:
for name, price in zip(names, prices):
# print(f"{name}\n{price}")
# print("-" * 50)
filename = str(name)+" "+str(price)+"\n"
split1 = filename.split(' / ')
cutted1 = split1.pop(0)
split2 = cutted1.split(": ")
cutted2 = split2.pop(1)
try:
two_prices = cutted2+" "+split1.pop(0)+"\n"
except IndexError:
two_prices = cutted2+"\n"
file.write(two_prices)但是在一些改变之后它在新的错误上停止-它不能通过给定的属性找到字符串，所以错误"IndexError：pop from empty list "出现...甚至soup.select("table tr td font>font")都没有帮助，就像它在" names "中所做的那样
2条答案
按热度按时间wrrgggsh1#
您只需要让CSS选择器更具体一些，以便只标识直接位于字体元素内部的链接（而不是向下几级）：
在页面底部添加链接指向单个项目而不是下一页/上一页链接的进一步标准也会有所帮助：
p8h8hvxi2#
您可以尝试下一个示例沿着分页
输出：