Wow, thanks everyone! This makes way more sense now.
I think I’ll start with Scrapy since it does both crawling and scraping. Tried BeautifulSoup first but got stuck on dynamic content—maybe Playwright’s the move?
Also, good call on checking robots.txt. Almost got blocked earlier lol.
One more Q: How do you handle sites with infinite scroll? Do you just mimic user behavior?
I think I’ll start with Scrapy since it does both crawling and scraping. Tried BeautifulSoup first but got stuck on dynamic content—maybe Playwright’s the move?
Also, good call on checking robots.txt. Almost got blocked earlier lol.
One more Q: How do you handle sites with infinite scroll? Do you just mimic user behavior?
