Newbie here too, but I learned the hard way that scraping all pages website robots.txt allows can still get you blocked. Some sites use Cloudflare or other protections that don’t care about robots.txt.
Try starting small—scrape a few pages first and see how the site reacts. Tools like Apify or Diffbot are nice if you don’t wanna deal with code.
And yeah, always be nice to the server. No one likes a hammering bot.
Try starting small—scrape a few pages first and see how the site reacts. Tools like Apify or Diffbot are nice if you don’t wanna deal with code.
And yeah, always be nice to the server. No one likes a hammering bot.