Skip to content

Commit 405bed4

Browse files
RPBotclaude
andcommitted
Sync web-scraping-with-scrapy-and-mongodb with the maintenance update of "Web Scraping With Scrapy and MongoDB"
- Replace the spider's def start_requests(self) with async def start(self); Scrapy removed Spider.start_requests() in 2.16.0, so on current Scrapy the method was never called and the errback on the initial request was silently dropped. - Re-pin requirements.txt to the tutorial's new dependency set (Scrapy 2.18.0, pymongo 4.17.0, Python 3.14). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent d867710 commit 405bed4

2 files changed

Lines changed: 34 additions & 37 deletions

File tree

‎web-scraping-with-scrapy-and-mongodb/books/books/spiders/book.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ class BookSpider(scrapy.Spider):
77
allowed_domains = ["books.toscrape.com"]
88
start_urls = ["https://books.toscrape.com/"]
99

10-
def start_requests(self):
10+
async def start(self):
1111
for url in self.start_urls:
1212
yield scrapy.Request(
1313
url, callback=self.parse, errback=self.log_error
Lines changed: 33 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -1,40 +1,37 @@
1-
attrs==23.2.0
2-
Automat==22.10.0
3-
certifi==2024.7.4
4-
cffi==1.16.0
5-
charset-normalizer==3.3.2
1+
attrs==26.1.0
2+
Automat==25.4.16
3+
brotli==1.2.0
4+
certifi==2026.7.22
5+
cffi==2.1.1
6+
charset-normalizer==3.5.1
67
constantly==23.10.4
7-
cryptography==43.0.0
8-
cssselect==1.2.0
8+
cryptography==50.0.1
9+
cssselect==1.5.0
910
defusedxml==0.7.1
10-
dnspython==2.6.1
11-
filelock==3.15.4
11+
dnspython==2.8.0
12+
filelock==3.32.7
1213
hyperlink==21.0.0
13-
idna==3.7
14-
incremental==22.10.0
15-
itemadapter==0.9.0
16-
itemloaders==1.3.1
17-
jmespath==1.0.1
18-
lxml==5.2.2
19-
packaging==24.1
20-
parsel==1.9.1
21-
Protego==0.3.1
22-
pyasn1==0.6.0
23-
pyasn1_modules==0.4.0
24-
pycparser==2.22
14+
idna==3.19
15+
incremental==24.11.0
16+
itemadapter==0.13.1
17+
itemloaders==1.4.0
18+
jmespath==1.1.0
19+
lxml==6.1.3
20+
packaging==26.3
21+
parsel==1.11.0
22+
Protego==0.6.2
23+
pycparser==3.0
2524
PyDispatcher==2.0.7
26-
pymongo==4.8.0
27-
pyOpenSSL==24.2.1
28-
queuelib==1.7.0
29-
requests==2.32.3
30-
requests-file==2.1.0
31-
Scrapy==2.11.2
32-
service-identity==24.1.0
33-
setuptools==71.1.0
34-
six==1.16.0
35-
tldextract==5.1.2
36-
Twisted==24.3.0
37-
typing_extensions==4.12.2
38-
urllib3==2.2.2
39-
w3lib==2.2.1
40-
zope.interface==6.4.post2
25+
pymongo==4.17.0
26+
pyOpenSSL==26.4.0
27+
queuelib==1.10.0
28+
requests==2.34.2
29+
requests-file==3.0.1
30+
Scrapy==2.18.0
31+
service-identity==26.1.0
32+
tldextract==5.3.2
33+
Twisted==26.4.0
34+
typing_extensions==4.16.0
35+
urllib3==2.8.0
36+
w3lib==2.4.1
37+
zope.interface==8.6

0 commit comments

Comments
 (0)