What’s the Best Way to Scrape Website Data Without Getting Blocked? or How Can I Ethically Scrape Web

14 Replies, 1752 Views

Yo, if you’re getting blocked while trying to scrape website data, slow down your requests! Like, add random delays between 2-10 seconds.

Tools? Scrapy + rotating proxies is my go-to. Free proxies suck tho, so maybe invest in Luminati or something.

Ethical scraping = don’t be a dick. Don’t take data that’s clearly private or behind a login.

CAPTCHAs are the worst. Sometimes changing your user agent helps, but no guarantees.

Messages In This Thread



Users browsing this thread: 1 Guest(s)