How can I scrape a whole website's robots.txt using BulkGPT?

7 Replies, 733 Views

Hi everyone!

Thanks for the helpful tips! 🙌 I’m definitely going to try out the suggestions about setting headers and logging requests.

I’ll also make sure to respect the robots.txt directives as I scrape whole website robot.txt how bulkgpt. If I run into any further questions or issues, I’ll be sure to ask!

Thanks again for your help! 😊

Messages In This Thread



Users browsing this thread: 1 Guest(s)