What’s the best way to start crawling with Python for web scraping? or How can I improve my crawling

"What’s the best way to start crawling with Python for web scraping?"

Hey folks! 👋 Newbie here trying to get into crawling with python. I’ve played around with requests and BeautifulSoup, but I feel like I’m missing something.

Any tips on libraries or frameworks to make it easier? Scrapy seems popular, but is it overkill for small projects? Also, how do you handle those pesky dynamic sites that load content with JS?

And uh… how do I not get blocked immediately? 😅

Thanks in advance!

---
*PS: If you’ve got any "gotchas" or rookie mistakes to avoid, lmk!*