Need help building a Python program to scrape website data – any tips or libraries to use? or What’s

20 Replies, 912 Views

Scrapy is overkill if you're just starting out. Stick with requests + BeautifulSoup for simple scraping.

For anti-blocking, try proxies (free ones like https://free-proxy-list.net but they’re hit or miss).

Also, check the site's `robots.txt` first—some sites straight-up ban scraping.

For structure, I usually do:
1. Fetch page
2. Parse data
3. Save to CSV/json

Keep it simple!

Messages In This Thread



Users browsing this thread: 1 Guest(s)