Hey! So, robots.txt is technically a suggestion, not a legal doc. Some sites *say* you can scrape all pages website robots.txt allows, but they might still block you if you hit them too hard.
Tools like Scrapy or BeautifulSoup can help, but yeah, slow crawling is key. Maybe set a delay of 2-3 secs between requests?
Also, check out ParseHub—it’s user-friendly for scraping without being aggressive.
Legally? Grey area. Even if robots.txt says yes, some sites have terms of service that forbid scraping. Tread carefully!
Tools like Scrapy or BeautifulSoup can help, but yeah, slow crawling is key. Maybe set a delay of 2-3 secs between requests?
Also, check out ParseHub—it’s user-friendly for scraping without being aggressive.
Legally? Grey area. Even if robots.txt says yes, some sites have terms of service that forbid scraping. Tread carefully!