我想创建一个自定义服务scrapyd,从scrapy项目,但我不断得到错误。
我创建了crawler/webservice.py
:
from scrapyd.webservice import WsResource
class BackInTime(WsResource):
def render_GET(self, txtrequest):
return {
'id': 'something works!'
}
然后修改了scrapyd.conf
,在[services]
标签下添加了backintime.json = crawler.webservice.BackInTime
行。
但是当我运行scrapyd
命令时,我总是得到错误:
...
File "c:\personal\virtualenvs\crawlie\lib\site-packages\scrapyd\app.py", line 39, in application
webservice = TCPServer(http_port, server.Site(webcls(config, app)), interface=bind_address)
File "c:\personal\virtualenvs\crawlie\lib\site-packages\scrapyd\website.py", line 33, in __init__
servCls = load_object(servClsName)
File "c:\personal\virtualenvs\crawlie\lib\site-packages\scrapy\utils\misc.py", line 46, in load_object
mod = import_module(module)
File "C:\Users\user\Anaconda3\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
builtins.ModuleNotFoundError: No module named 'crawler'
Failed to load application: No module named 'crawler'
尝试遵循以前的一些解决方案,但没有任何成功:
- https://groups.google.com/forum/#!searchin/scrapy-users/webservice/scrapy-users/czu8YeYFvao/8GDHMJbi56wJ
- How to add a new service to scrapyd from current project
- Implementing own scrapyd service
我应该如何添加新服务以便scrapyd可以导入它?
1条答案
按热度按时间jogvjijk1#
尝试将应用的导入模块添加到环境变量
PYTHONPATH
。在我的Dockerfile
中,它看起来像这样:在您的本地计算机上,它可能看起来像:
然后运行Scrapyd。