Scrapy is actually pretty lightweight once you get the hang of it, and it handles a lot of the messy stuff for you (like retries, throttling).
But if you're just dipping your toes into crawling with python, maybe stick with requests + bs4 first. For dynamic content, Puppeteer (JS) is great, but if you're Python-only, Pyppeteer is a decent alternative.
Oh, and always—ALWAYS—add delays between requests. Sites hate getting bombarded.
But if you're just dipping your toes into crawling with python, maybe stick with requests + bs4 first. For dynamic content, Puppeteer (JS) is great, but if you're Python-only, Pyppeteer is a decent alternative.
Oh, and always—ALWAYS—add delays between requests. Sites hate getting bombarded.