调用text.strip()
时出现以下错误:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-278-135ac185ec3f> in <module>
20 if isinstance(b, Tag):
21
---> 22 location = [a.text.strip() for a in b.find('span', attrs = {'class': 'location'})]
23 job_title = [a.text.strip() for a in b.find('a', attrs = {'data-tn-element':'jobTitle'})]
24
TypeError: 'NoneType' object is not iterable
请看下面的代码:
import requests
from bs4 import BeautifulSoup, NavigableString, Tag, Comment
import pandas as pd
df = pd.DataFrame(columns=["location", 'company', 'job_title', 'salary'])
for start in range(1,100,10):
url = 'https://www.indeed.com/jobs?q=python+sql&l=San+Francisco&start={}'
#format url above to request the various search pages
new_url = url.format(start)
#conducting a request of the stated URL above:
page = requests.get(new_url)
#specifying a desired format of “page” using the html parser - this allows python to read the various components of the page, rather than treating it as one long string.
soup = BeautifulSoup(page.text, 'html.parser')
#loop through the tag elements
for b in soup.find_all(name = 'div', attrs={'class':'jobsearch-SerpJobCard'}):
print(type(b))
if isinstance(b,NavigableString):
continue
if isinstance(b, Tag):
location = [a.text.strip() for a in b.find('span', attrs = {'class': 'location'})]
job_title = [a.text.strip() for a in b.find('a', attrs = {'data-tn-element':'jobTitle'})]
try:
company = [a.text.strip() for a in b.find('span', attrs = {'class':'company'})]
except:
company = 'NA'
try:
salary = [a.text.strip() for a in b.find('span', attrs = {'class' : 'salaryText'}).find('nobr')]
except:
salary = 'NA'
df = df.append({"location":location,"company":company, "job_title": job_title, "salary": salary}, ignore_index=True)
2条答案
按热度按时间h5qlskok1#
找不到,因为页面上没有class属性设置为'location'的。有class属性设置为'location'的。以下是我的修改版本,仍然是不完美的,因为有些位置没有被抓取。一个想法是跳过那些没有工作或位置,如果这两个参数是必要的。您可以通过将except操作替换为将“NA”赋值给
continue
来完成此操作ogsagwnx2#
您将需要添加对None值的检查,如果没有找到元素,
find
将返回None。