Hey! I totally feel you on the overwhelming part. When I first started learning how do you create a web crawler, I was like, "Where do I even begin?" Python is definitely the way to go, especially with libraries like Scrapy. It handles a lot of the heavy lifting for you.
For dynamic content, Puppeteer is a solid alternative to Selenium. It’s faster and works great with JavaScript-heavy sites. Also, check out `requests-html` if you want something lightweight.
As for respecting crawl limits, I use `time.sleep()` to add delays between requests. It’s not perfect, but it helps avoid hammering servers. Good luck!
For dynamic content, Puppeteer is a solid alternative to Selenium. It’s faster and works great with JavaScript-heavy sites. Also, check out `requests-html` if you want something lightweight.
As for respecting crawl limits, I use `time.sleep()` to add delays between requests. It’s not perfect, but it helps avoid hammering servers. Good luck!