What's the best way to start crawling with Python for web scraping? or How can I improve my crawling

OP reply:
Wow, thanks for all the tips! Tried Playwright based on your suggestions and it’s way smoother than Selenium.

Still getting blocked sometimes though—guess I’ll look into proxies. Anyone got a fav free proxy list?

Also, Scrapy seems intimidating but maybe I’ll give it a shot next weekend. Cheers!